Runner removal 'update tags' query can become blocked in the database
Sentry: https://sentry.gitlap.com/gitlab/gitlabcom/issues/31207/
Slack conversation: https://gitlab.slack.com/archives/C101F3796/p1495217999718237
When removing a runner, at some point we execute a query like `UPDATE "tags" SET "taggings_count" = COALESCE("taggings_count", 0) + 1 WHERE "tags"."id" = 846`. This query can be blocked by a delete query for pipelines such as `DELETE FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 4614038 AND "ci_pipelines"."lock_version" IS NULL`.
We confirmed with a GitLab.com user that they did not delete any projects prior to attempting to delete the runner. It seems that another user's action somehow causes issues with the runner delete.
@stanhu thinks "we should see that removing a runner with the same tag as another project would cause `taggings_count` to be updated"
cc/ @ayufan
issue