Deleting App Insight doesn't delete web tests
I deleted an app insight (AI1) which contained a webtest. I then created an app insight (AI2) that pointed to the same deployment. I noticed WT1 was still running as it was reported in AI2. The only way to delete this was to re-add AI1, and the delete the webtest.
Michael Sync commented
The web tests will automatically be deleted after 30 mins or so.
I accidentally delete the app insight before deleting the web tests. The delete button for webtests are all grayed out and it shows a small exclamation mark on web test icon. After 20 or 30 mins, those web tests are gone automatically.