Store WAL locations in concurrency limiter
What does this MR do and why?
In ConcurrencyLimit::ResumeWorker, we're resuming execution of previously deferred jobs (read more here). During high amount of deferred jobs, we noticed the execution of ResumeWorker's biggest bottleneck is querying DB's WAL LSN on the sidekiq client middleware. This query should be very fast under normal circumstances, but when large amount of jobs are querying at the same time, we saw high wait time from Sidekiq process to acquire a connection from the connection pool.
To optimize this, we could skip the LSN queries on the resumed jobs.
Before the job was deferred, the WAL locations was already queried in the previous client middleware. This MR stores the wal_locations field in the "concurrency limit"'s queue. When the job is resumed (via worker_klass.bulk_perform_async), the wal_locations alongside with other fields such as concurrency_limit_resume and contexts are stored in the RequestStore. These fields are then merged back in this middleware. Finally, the load balancing middleware wouldn't have to query the WAL LSN again since the wal_locations field is present.
More discussion on skipping the query: gitlab-com/gl-infra/data-access/durability/team#9 (comment 2232399012)
Related to gitlab-com/gl-infra/data-access/durability/team#18
References
Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
| Before | After |
|---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.