Add end-to-end latency metric for namespace transfers

What does this MR do and why?

Follow-up to !246573 (merged), which added transfer counters and an execution-time histogram.

The issue listed four observability items. This MR handles them as follows:

# Item How it is handled
1 queue_wait_s Removed from the Namespaces::TransferLogging payload.
2 Dashboard/alert on scheduling latency for Projects::TransferWorker Not in this MR.
3 End-to-end latency New histogram, described below.
4 deferred_count No code change needed.

1. queue_wait_s

No caller ever set it. Sidekiq's structured logger already logs queue_duration_s and scheduling_latency_s for every job, so no custom field is needed.

2. Dashboard and alert

This belongs in the runbooks and metrics catalog, not in this repository. It needs a separate follow-up, so the issue stays open. That is why this MR says "Related to" and not "Closes".

3. End-to-end latency

Adds the Prometheus histogram gitlab_namespace_transfer_end_to_end_duration_seconds{namespace_type}.

  • Projects::TransferWorker and Namespaces::Groups::TransferWorker observe it when a transfer completes successfully.
  • It measures from state_metadata['transfer_scheduled_at'] to completion. The transfer service sets that timestamp when it schedules the transfer, before it enqueues the worker. The value therefore includes queue wait and Sidekiq deferrals.
  • The worker reads the timestamp before complete_transfer!, because a before_transition callback (clear_transfer_data) clears it.
  • If the timestamp can't be parsed, the metric is skipped rather than raising. Otherwise a completed transfer would be logged as failed and retried.
  • Buckets run from 1 second to 24 hours. Database health check deferrals back off up to 30 minutes each, and a backlog can last hours.
  • Caveat: a failed attempt clears transfer_scheduled_at and the Sidekiq retry schedules again. The value is measured from the most recent schedule, not the first.

4. deferred_count

No code change. The Sidekiq done log for the run that finally executes already includes deferred_count and deferred_by. The structured logger copies the job hash when the job starts, before SkipJobs removes the deferred flag.

Example Kibana filter:

json.class: "Projects::TransferWorker" AND json.job_status: "done" AND json.deferred_count: *

I verified this locally with a spec against Gitlab::SidekiqLogging::StructuredLogger.

References

Screenshots or screen recordings

N/A, backend-only change.

How to set up and validate locally

  1. Run the specs:

    bundle exec rspec spec/lib/gitlab/metrics/transfers_spec.rb spec/workers/concerns/namespaces/transfer_logging_spec.rb spec/workers/projects/transfer_worker_spec.rb spec/workers/namespaces/groups/transfer_worker_spec.rb
  2. Optional: transfer a project in GDK and check /-/metrics for gitlab_namespace_transfer_end_to_end_duration_seconds.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Shubham Kumar

Merge request reports

Loading
Loading