Add Prometheus counter for stuck Duo Workflow sessions cleanup

Summary

Adds a Prometheus counter to CleanStuckWorkflowsService to track the number of stuck Duo Workflow sessions cleaned up, broken down by original status label (created or running).

This enables alerting on a high rate of sessions that were created but never started — the scenario described in the incident linked in the issue, where CLI initialization failures left sessions permanently in created status.

The counter can be used to alert on:

rate(gitlab_duo_workflow_stuck_workflows_cleaned_total{status="created"}[5m])

What changed

  • Added STUCK_WORKFLOWS_COUNTER constant to CleanStuckWorkflowsService using Gitlab::Metrics.counter
  • Incremented the counter per workflow before calling w.drop, while status_name still reflects the original status
  • Added a dedicated Prometheus counter spec context with :prometheus tag to verify label values

Verification steps

  1. Make sure sidekiq metrics are enabled
 gdk config set gitlab.rails_background_jobs.sidekiq_exporter_enabled true && gdk reconfigure
  1. Create a workflow with updated at in the past
w = ::Ai::DuoWorkflows::Workflow.new(project_id: 1000000, user_id: 1, goal: "my goal", updated_at: 3.days.ago)
w.save
  1. Call the worker
Ai::DuoWorkflows::FailStuckWorkflowsWorker.perform_async
  1. Visit http://gdk.test:3807/metrics
  2. You should see gitlab_duo_workflow_stuck_workflows_cleaned_total incremented

Screenshot

Screenshot_2026-07-29_at_22.57.33

References

Edited by Halil Coban

Merge request reports

Loading
Loading