Runner removal 'update tags' query can become blocked in the database
Sentry: https://sentry.gitlap.com/gitlab/gitlabcom/issues/31207/ Slack conversation: https://gitlab.slack.com/archives/C101F3796/p1495217999718237
When removing a runner, at some point we execute a query like UPDATE "tags" SET "taggings_count" = COALESCE("taggings_count", 0) + 1 WHERE "tags"."id" = 846
. This query can be blocked by a delete query for pipelines such as DELETE FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 4614038 AND "ci_pipelines"."lock_version" IS NULL
.
We confirmed with a GitLab.com user that they did not delete any projects prior to attempting to delete the runner. It seems that another user's action somehow causes issues with the runner delete.
@stanhu thinks "we should see that removing a runner with the same tag as another project would cause taggings_count
to be updated"
cc/ @ayufan