As noted in https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/167, we have observed that the AuthorizedProjectWorker is performing poorly. In the slowest cases, it can take around 30s to execute. https://log.gprd.gitlab.net/goto/3762fa17264bc3415277293f1f8a5fff In the corresponding issue in the Scalability Tracker, we are looking at how this job runs on GitLab.com, but we need some assistance from gitlab~10046105 to asses if there are any improvements that can be made to the job itself. # Exit Criteria ## Reduced number of jobs Specialized workers are used for authorization updates generating only a few number of jobs per event. The current approach generates a job for each member leading to [huge spikes in the queue length](https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail?viewPanel=922970012&orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-queue=post_receive&var-queue=authorized_projects&from=now-7d&to=now). Target: Reduce the number of mutating `AuthorizedProjectsWorker` jobs to 0 https://log.gprd.gitlab.net/app/lens#/edit/a2e77d70-a734-11eb-966b-2361593353f9?_g=h@d01ba18 ## Significantly reduced resource usage Periodic recalculation / verification of permissions runs on replica and does not require significant resources on primary DB. - currently [38% of all the rows returned by the server](https://prometheus-db.gprd.gitlab.net/graph?g0.expr=increase(pg_stat_statements_rows%7Btype%3D%22patroni%22%2C%20fqdn%3D%22patroni-03-db-gprd.c.gitlab-production.internal%22%2C%20queryid%3D%225005987689040900854%22%7D%5B1d%5D)%0A%2F%20ignoring%20(queryid)%0Asum%20without(queryid)%20(increase(pg_stat_statements_rows%7Btype%3D%22patroni%22%2C%20fqdn%3D%22patroni-03-db-gprd.c.gitlab-production.internal%22%7D%5B1d%5D))&g0.tab=0&g0.stacked=0&g0.range_input=4w&g0.end_input=2021-04-06%2009%3A00%3A00&g0.moment_input=2021-04-06%2009%3A00%3A00) is queried by project authorizations update / verification: https://gitlab.com/gitlab-org/database-team/team-tasks/-/issues/142 - recalculation contributes to database cpu spikes and is [currently in the top 10 queries](7232084447659837857): https://gitlab.com/gitlab-org/gitlab/-/issues/292242 and https://gitlab.com/gitlab-org/gitlab/-/issues/325688. Target: Reduce average rows returned by database from 38% -> 23% https://prometheus-db.gprd.gitlab.net/graph?g0.expr=sum%20by(fqdn)%20(rate(pg_stat_statements_rows%7Btype%3D%22patroni%22%2C%20queryid%3D~%22-8911006510108689902%22%7D%5B4h%5D))%0A%2F%20sum%20by(fqdn)%20(rate(pg_stat_statements_rows%7Btype%3D%22patroni%22%7D%5B4h%5D))&g0.tab=0&g0.stacked=0&g0.range_input=1w