Skip to content

Delete invalid epic_issue records

Jarka Košanová requested to merge 339514-delete-invalid-records into master

What does this MR do?

In https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/1626 we fixed the problem that some issues were assigned to an epic while the issues projects did not belong under the epic group (or its descendants). Now the issues are not displayed anymore so the security problem was fixed.

In this MR we delete those invalid records.

Follow-up

In the next MR validation on EpicIssue model will be added.

Database

Schedule migration

== 20211023102243 ScheduleDeleteInvalidEpicIssues: migrating ==================
-- Scheduled 1 DeleteInvalidEpicIssues jobs with a maximum of 10000 records per batch and an interval of 120 seconds.

The migration is expected to take at least 120 seconds. Expect all jobs to have completed after 2021-10-26 20:43:42 UTC."
== 20211023102243 ScheduleDeleteInvalidEpicIssues: migrated (3.6027s) =========

Queries with the execution plans

SELECT "epics"."id" FROM "epics" ORDER BY "epics"."id" ASC LIMIT 1

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/7428/commands/26350

SELECT "epics"."id" FROM "epics" WHERE "epics"."id" >= 1 ORDER BY "epics"."id" ASC LIMIT 1 OFFSET 10000

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/7428/commands/26351

SELECT MIN("epics"."id"), MAX("epics"."id") FROM "epics" WHERE "epics"."id" >= 1 AND "epics"."id" < 10001

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/7428/commands/26352

Followed by the next batch with: SELECT "epics"."id" FROM "epics" WHERE "epics"."id" >= 10001 ORDER BY "epics"."id" ASC LIMIT 1 OFFSET 10000

Migration

Queries

Getting epics
SELECT "epics".* FROM "epics" WHERE "epics"."id" BETWEEN 1 AND 1001 ORDER BY "epics"."id" ASC LIMIT 1000

Query plan: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/7428/commands/26354

Getting epic associations
SELECT "epic_issues".* FROM "epic_issues" WHERE "epic_issues"."epic_id" IN ()
SELECT "issues".* FROM "issues" WHERE "issues"."id" IN ()
SELECT "projects".* FROM "projects" WHERE "projects"."id" IN () 
Getting group & ancestors for every epic
WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."id" = 9970)
UNION
(SELECT "namespaces".* FROM "namespaces", "base_and_ancestors" WHERE "namespaces"."id" = "base_and_ancestors"."parent_id")) SELECT "namespaces".* FROM "base_and_ancestors" AS "namespaces"

Query plan: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/7056/commands/24921

Deleting epic_issue records

DELETE FROM "epic_issues" WHERE "epic_issues"."id" IN (18, 20, 22)

Counts

  • Epics with epic issues: around 73 500
  • All epics: around 147 000
  • All epic issues: almost 750 000

MR acceptance checklist

Related to #339514

Edited by Jarka Košanová

Merge request reports

Loading