LooseForeignKeys::CleanupWorker is falling behind while processing CI LFK_deleted_records
This issue happened before. See issue. It's happening again now. LooseForeignKeys::CleanupWorker
cannot catch up on the number of items on ci
database.
To temporarily mitigate this issue, we can enable OPS feature flag loose_foreign_keys_turbo_mode_ci
, so that the clean up worker catches up. But that probably doesn't solve the original problem. There must be some slow queries.
References:
- https://log.gprd.gitlab.net/app/r?l=DISCOVER_APP_LOCATOR&v=8.12.1&lz=N4IgjgrgpgTgniAXKSsGJAKwM4HsB2AdFAB4AuMAhoQDa67ZQD6AZrjFAJYDm%2BTA1lDjYmAYxpRK%2BCAAcmAd3aCYhbGUplshUQXxRRZTgUQACAFQmQAGhA0p3CJW5QkIftHggAvjbwwySADagVh4RIYAti42ACZQ2KIgALpJNjo0EBH42EGhBIQxEFSGBNZ5RIXFRnw5Njj54pTYteXE5FS09Iys7Fy8AkIi4pLScoowyqrqmtq6%2BiX4yTac%2BHEkrgCCAOoAchsAQgAiAEryO4cQAIoAEgAqEQBMZNcAbDRlK2SwAG6U7xiUCBkXBlFicGhfGA5RCBVIgAAWnDiAGF4ZR%2FEgWH9GLENJQAGqcKDyACSMU2uwOJzOFxu9yer3eNkiUGO9hcyBALBguAirnwuHkAFoAKzwsrA%2FmC7w2Djc%2BLwkn4SG%2Ff6gVXQJAvAAMuu1NhkgMYSAo0C8XiAA
- 19 partitions on
ci
databaseloose_foreign_keys_deleted_records
table. https://postgres.ai/console/gitlab/gitlab-production-ci/sessions/33826/commands/103991 - Original issue: #419119 (closed)
Open related issue:
- Alerting: #419580 (closed)
Edited by Omar Qunsul