Improve the performance of the Projects::FetchStatisticsIncrementService
The workhorse_auth_api Apdex (This is the indicator that has had some huge drops lately, for example:
Looking at the logs for the same timeframe, we can see that p99 database duration-peaks seem to correlate for the git-upload-pack endpoint:
https://log.gprd.gitlab.net/goto/3e4a237bbdbe6c76da4fcfffd5502203
@andrewn was sampling postgres to investigate database locks on the project_daily_statistics
, using the info from marginalia to tie the queries to endpoints and noticed that there were a lot of queries for git_upload_pack
marked as idle
: gdoc with raw data
Which makes us think we should improve the way we track these fetch statistics
Proposals
-
Remove the tracker all together: we only increment this counter for git-https, but not over SSH. Which means the counter doesn't paint an entirely accurate picture. So perhaps this counter, that is only accessible through this API endpoint, could be removed?
-
Only periodically increment a counter in the database, while keeping a live counter in Redis. When incrementing the counter, we schedule a sidekiq job with a delay of a few minutes that can be deduplicated (using
deduplicate :until_executing, including_scheduled: true
.