Improve the performance of the Projects::FetchStatisticsIncrementService
The workhorse_auth_api Apdex (This is the indicator that has had some huge drops lately, for example:
Looking at the logs for the same timeframe, we can see that p99 database duration-peaks seem to correlate for the git-upload-pack endpoint:
@andrewn was sampling postgres to investigate database locks on the
project_daily_statistics, using the info from marginalia to tie the queries to endpoints and noticed that there were a lot of queries for
git_upload_pack marked as
idle: gdoc with raw data
Which makes us think we should improve the way we track these fetch statistics
Remove the tracker all together: we only increment this counter for git-https, but not over SSH. Which means the counter doesn't paint an entirely accurate picture. So perhaps this counter, that is only accessible through this API endpoint, could be removed?
Only periodically increment a counter in the database, while keeping a live counter in Redis. When incrementing the counter, we schedule a sidekiq job with a delay of a few minutes that can be deduplicated (using
deduplicate :until_executing, including_scheduled: true.