Loading
Add Prometheus counter for stuck Duo Workflow sessions cleanup
Summary
Adds a Prometheus counter to CleanStuckWorkflowsService to track the number of stuck Duo Workflow sessions cleaned up, broken down by original status label (created or running).
This enables alerting on a high rate of sessions that were created but never started — the scenario described in the incident linked in the issue, where CLI initialization failures left sessions permanently in created status.
The counter can be used to alert on:
rate(gitlab_duo_workflow_stuck_workflows_cleaned_total{status="created"}[5m])What changed
- Added
STUCK_WORKFLOWS_COUNTERconstant toCleanStuckWorkflowsServiceusingGitlab::Metrics.counter - Incremented the counter per workflow before calling
w.drop, whilestatus_namestill reflects the original status - Added a dedicated
Prometheus counterspec context with:prometheustag to verify label values
Verification steps
- Make sure sidekiq metrics are enabled
gdk config set gitlab.rails_background_jobs.sidekiq_exporter_enabled true && gdk reconfigure- Create a workflow with updated at in the past
w = ::Ai::DuoWorkflows::Workflow.new(project_id: 1000000, user_id: 1, goal: "my goal", updated_at: 3.days.ago)
w.save- Call the worker
Ai::DuoWorkflows::FailStuckWorkflowsWorker.perform_async- Visit http://gdk.test:3807/metrics
- You should see
gitlab_duo_workflow_stuck_workflows_cleaned_totalincremented
Screenshot
References
Edited by Halil Coban
