Add end-to-end latency metric for namespace transfers
What does this MR do and why?
Follow-up to !246573 (merged), which added transfer counters and an execution-time histogram.
The issue listed four observability items. This MR handles them as follows:
| # | Item | How it is handled |
|---|---|---|
| 1 | queue_wait_s |
Removed from the Namespaces::TransferLogging payload. |
| 2 | Dashboard/alert on scheduling latency for Projects::TransferWorker |
Not in this MR. |
| 3 | End-to-end latency | New histogram, described below. |
| 4 | deferred_count |
No code change needed. |
1. queue_wait_s
No caller ever set it. Sidekiq's structured logger already logs queue_duration_s and scheduling_latency_s for every job, so no custom field is needed.
2. Dashboard and alert
This belongs in the runbooks and metrics catalog, not in this repository. It needs a separate follow-up, so the issue stays open. That is why this MR says "Related to" and not "Closes".
3. End-to-end latency
Adds the Prometheus histogram gitlab_namespace_transfer_end_to_end_duration_seconds{namespace_type}.
Projects::TransferWorkerandNamespaces::Groups::TransferWorkerobserve it when a transfer completes successfully.- It measures from
state_metadata['transfer_scheduled_at']to completion. The transfer service sets that timestamp when it schedules the transfer, before it enqueues the worker. The value therefore includes queue wait and Sidekiq deferrals. - The worker reads the timestamp before
complete_transfer!, because abefore_transitioncallback (clear_transfer_data) clears it. - If the timestamp can't be parsed, the metric is skipped rather than raising. Otherwise a completed transfer would be logged as failed and retried.
- Buckets run from 1 second to 24 hours. Database health check deferrals back off up to 30 minutes each, and a backlog can last hours.
- Caveat: a failed attempt clears
transfer_scheduled_atand the Sidekiq retry schedules again. The value is measured from the most recent schedule, not the first.
4. deferred_count
No code change. The Sidekiq done log for the run that finally executes already includes deferred_count and deferred_by. The structured logger copies the job hash when the job starts, before SkipJobs removes the deferred flag.
Example Kibana filter:
json.class: "Projects::TransferWorker" AND json.job_status: "done" AND json.deferred_count: *I verified this locally with a spec against Gitlab::SidekiqLogging::StructuredLogger.
References
- Follow-up to !246573 (merged)
- Related to #623298 (closed)
Screenshots or screen recordings
N/A, backend-only change.
How to set up and validate locally
-
Run the specs:
bundle exec rspec spec/lib/gitlab/metrics/transfers_spec.rb spec/workers/concerns/namespaces/transfer_logging_spec.rb spec/workers/projects/transfer_worker_spec.rb spec/workers/namespaces/groups/transfer_worker_spec.rb -
Optional: transfer a project in GDK and check
/-/metricsforgitlab_namespace_transfer_end_to_end_duration_seconds.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.