Add correlation_duration_s to Sidekiq and request logs
What does this MR do and why?
Sidekiq job logs and request logs now carry correlation_duration_s, the time in seconds since the root request or job of the call chain started. Chains of workers that keep running for hours, or loop back on themselves, can then be found in Kibana with a filter like json.correlation_duration_s > 3600.
A web request seeds root_started_at in Gitlab::ApplicationContext. Sidekiq jobs inherit it through Labkit the same way root_caller_id travels today, and jobs enqueued without a request, such as cron jobs, seed it at enqueue time. The structured logger derives the duration from it on the start and done lines. Decoding the timestamp in correlation_id would miss cron chains, because labkit-ruby still generates random hex IDs for them.
The new context key stays out of the Sidekiq queue deletion API, so the GraphQL schema does not change.
References
- Plan and discussion: https://gitlab.com/gitlab-org/gitlab/-/issues/415327
- Investigation this unblocks: https://gitlab.com/gitlab-org/gitlab/-/issues/352038
Screenshots or screen recordings
No UI changes.
How to set up and validate locally
-
Push a commit to any project in GDK, so that
PostReceiveruns and enqueues further jobs. -
Read the Sidekiq log:
tail -n 200 log/sidekiq.log | jq -c 'select(.correlation_duration_s != null) | {class, correlation_id, job_status, correlation_duration_s}' -
PostReceiveand the jobs it enqueued share onecorrelation_id, andcorrelation_duration_sgrows along the chain. -
Wait for any cron job to run. Its log lines also carry the field, measured from when the cron poller enqueued it.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.