Resolve "Allow periodic project authorization refresh jobs to use database replica for reads" [RUN ALL RSPEC] [RUN AS-IF-FOSS]
What does this MR do?
For #292443 (closed), and is implementing the plan that was roughly mentioned in !54836 (comment 515219453)
Premise
Currently we enqueue jobs in batched fashion via AuthorizedProjectUpdate::PeriodicRecalculateWorker
using a cron job
The purpose of this job is to check if a specific user has any mismatch in their project_authorization
records, and if so, fix this discrepancy by adding or removing the necessary records.
Problem
Since we have many users on GitLab.com, the number of these jobs are high, and all these jobs currently read and write project_authorizations
data form the primary database, which is increasing the load on the database.
Approach
With this MR, we are trying to reduce the load on the primary database, by off-loading at least the "read" part of this process to the replica database.
With this change, for a specific user:
- The worker now asks the replica database - "Does the user require a project authorization refresh?"
- Based on the data currently in the replica db for this user, it returns a
true
orfalse
. - If
true
, we enqueue a new job that completely runs on the primary database that checks (again) if the user needs a refresh based on the data in the primary database, and if so completes it by adding or removing records from the primary database.
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. -
I have not included a changelog entry because the change is behind a feature flag.
-
-
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team