Skip to content

Use CounterAttribute for ProjectDailyStatistic

Baptiste Lalanne requested to merge gitlab-community/gitlab:426270-counter-attribute-project-daily-statistic into master
  • Please check this box if this contribution uses AI-generated content as outlined in the GitLab DCO & CLA

What does this MR do and why?

👋 This MR make use of the CounterAttribute as suggested by @fabiopitino for the daily fetch count.

During peak hours, an extreme amount of fetches will try to update the count on the same tuple (project_id, date) which can make the LockManager of Postgres to make those queries wait longer which can cause other queries to fail and would create 500s. INSERTING with the Counter Attribute is a perfect improvement of the potential surge of INSERT that pipelines with high concurrency can caused.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

image

gitlabhq_development=# SELECT "project_daily_statistics".* FROM "project_daily_statistics" WHERE "project_daily_statistics".id=39
;
 id | project_id | fetch_count |    date
----+------------+-------------+------------
 39 |          3 |           2 | 2024-04-14
(1 row)

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested. There might easier ways but to debug that's what I did:

  1. Run couple CI jobs in a project
  2. Look into Redis for the key
  3. Look into Sidekiq admin to see the schedule task to flush the counter.
  4. Look into the DB to make sure the flush worked: SELECT "project_daily_statistics".* FROM "project_daily_statistics" WHERE "project_daily_statistics"."project_id" = 3 AND "project_daily_statistics"."date" = '2024-04-11' LIMIT 1;

Closes #426270

Edited by Baptiste Lalanne

Merge request reports