Measure CI processing delay

What does this MR do and why?

Track the time it takes for a job to leave the created status after all of its dependencies have finished, and observe it using LabKit's user experience SLI SDK. The new user experience is called ci_job_processing_delay. The tracking is behind the ops feature flag ci_observe_job_processing_delay, scoped to the current request.

References

Part of #592819

Screenshots or screen recordings

Before	After

How to set up and validate locally

1. In rails console enable the feature flag

   Feature.enable(:ci_observe_job_processing_delay)

2. In your terminal, tail the user experience SLI log for matching lines:

   tail -f log/user_experience_slis.log | grep ci_job_processing_delay

3. Run a few pipelines and observe the logs. You should see one "checkpoint":"start" event and one "checkpoint":"end" per CI job.

Example:

{
  "severity": "INFO",
  "time": "2026-04-18T03:34:56.626Z",
  "correlation_id": "01KPFAFRQ9HK7A4G2Z2JBK9S15",
  "meta.caller_id": "Ci::InitialPipelineProcessWorker",
  "meta.remote_ip": "172.16.123.1",
  "meta.feature_category": "continuous_integration",
  "meta.user": "root",
  "meta.gl_user_id": 1,
  "meta.project": "root/test",
  "meta.root_namespace": "root",
  "meta.client_id": "user/1",
  "meta.root_caller_id": "POST /api/:version/projects/:id/merge_requests/:merge_request_iid/pipelines",
  "checkpoint": "start",
  "user_experience_id": "ci_job_processing_delay",
  "feature_category": "continuous_integration",
  "urgency": "async_fast",
  "start_time": "2026-04-18T03:34:56.372Z",
  "end_time": "2026-04-18T03:34:56.626Z",
  "elapsed_time_s": 0.253754,
  "urgency_threshold_s": 15
}
{
  "severity": "INFO",
  "time": "2026-04-18T03:34:56.626Z",
  "correlation_id": "01KPFAFRQ9HK7A4G2Z2JBK9S15",
  "meta.caller_id": "Ci::InitialPipelineProcessWorker",
  "meta.remote_ip": "172.16.123.1",
  "meta.feature_category": "continuous_integration",
  "meta.user": "root",
  "meta.gl_user_id": 1,
  "meta.project": "root/test",
  "meta.root_namespace": "root",
  "meta.client_id": "user/1",
  "meta.root_caller_id": "POST /api/:version/projects/:id/merge_requests/:merge_request_iid/pipelines",
  "checkpoint": "end",
  "user_experience_id": "ci_job_processing_delay",
  "feature_category": "continuous_integration",
  "urgency": "async_fast",
  "start_time": "2026-04-18T03:34:56.372Z",
  "end_time": "2026-04-18T03:34:56.626Z",
  "elapsed_time_s": 0.253754,
  "urgency_threshold_s": 15
}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.