Compliance center status is not updated for all frameworks

Summary

The compliance center status is not being properly updated for all frameworks consistently. The ComplianceManagement::FrameworkEvaluationSchedulerWorker is intermittently failing due to PG timeouts, though it has failed multiple times in a row over the last ~2 days.

As this worker operates in batches, this is not impacting all projects. We are seeing that the ComplianceManagement::ProjectComplianceEvaluatorWorker is kicked off every 12 hours, it just isn't processing all of the expected frameworks due to the scheduler worker failing.

Steps to reproduce

N/A

What is the current bug behavior?

Not all compliance frameworks are properly evaluated on their 12 hour schedule due to PG timeouts with the ComplianceManagement::FrameworkEvaluationSchedulerWorker worker.

What is the expected correct behavior?

All compliance frameworks are properly evaluated on their 12 hour schedule.

Relevant logs and/or screenshots

The error ComplianceManagement::FrameworkEvaluationSchedulerWorker is returning in sidekiq:

PG::QueryCanceled
ActiveRecord::QueryCanceled
PG::QueryCanceled: ERROR:  canceling statement due to statement timeout

The exception SQL can be found in the log entries below.

Executions of ComplianceManagement::FrameworkEvaluationSchedulerWorker over the last 7 days - the last two have failed due to the PG timeout:

• https://log.gprd.gitlab.net/app/r/s/fNDom

Executions of ComplianceManagement::ProjectComplianceEvaluatorWorker over the last 2 days - it is regularly running on a 12 hour interval:

• https://log.gprd.gitlab.net/app/r/s/xZHqH

A compliance framework ID that has not been updated since the back to back failures have occurred:

• https://log.gprd.gitlab.net/app/r/s/4dmnT

Output of checks

This bug happens on GitLab.com

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.