Draft: Post GitLab Duo root cause analysis for Product Security pipelines

Jennifer Lirequested to merge
prodsec-pipeline-failure-rca into master

Problem

Pipeline failures in the Product Security tooling repos reach #sec-prod-sec-eng-automation-failures through GitLab's pipelines-email integration, pointed at a Slack email-to-channel address. That alert reports that a pipeline broke but nothing about why, so triaging it always starts with opening the job log by hand.

GitLab already ships the feature that answers this: Duo Root Cause Analysis, the /troubleshoot chat tool scoped to a Ci::Build. This MR wires it to the pipeline.failed webhook so the explanation shows up in the channel on its own.

I first looked at having triage-ops post @GitLab why did the pipeline <url> fail and let the GitLab for Slack app answer. That does not work: Duo in Slack is an Experiment behind the default-off slack_duo_agent flag, it requires a Slack account linked to a GitLab account (a bot has none, and the authorize prompt is ephemeral), and it runs the Developer Flow over channel history rather than the CI root-cause tool.

Changes

Triage::PipelineFailure::RootCauseAnalysis issues an aiAction mutation scoped to the failed build, then polls aiMessages for the assistant reply. GitLab reads the job log server-side, so job logs never pass through triage-ops. Polling is bounded at 60s, well under the 110s SUCKER_PUNCH_SHUTDOWN_TIMEOUT. Any failure — blank token, refusal, message errors, timeout, non-200 — returns nil instead of raising.

Triage::ProductSecurity::PipelineFailureRca reacts to pipeline.failed for projects under gitlab-com/gl-security/ and gitlab-private/gl-security/, skipping merge request and triggered pipelines. It matches on namespace prefix rather than an explicit project list, since webhook configuration is already the real gate. When no analysis comes back it posts nothing, so it never duplicates the existing email alert.

I kept this as a standalone processor rather than another PipelineFailure::Config class: PipelineHealth::PipelineFailureManagement is built around incident creation and escalation, none of which applies here, and it is the critical master-broken path.

It authenticates with a new GITLAB_SECURITYBOT_API_TOKEN rather than the shared triage-ops bot token, because troubleshoot_job_with_ai resolves to can?(:read_build) & troubleshoot_job_cloud_connector_authorized & troubleshoot_job_with_ai_authorized. That access belongs to Product Security's own gitlab-securitybot identity, which already holds Reporter on the in-scope projects. Leaving the variable unset disables the analysis.

Validation

  • New specs pass: 30 examples, 0 failures. RuboCop clean on all four new files.
  • Full suite: 5212 examples, 6 failures, all in spec/processor/runway_migration_test_spec.rb. They reproduce identically on a clean master (they need GITLAB_BOT_API_TOKEN_ON_COM locally) and are unrelated to this change.
  • glab ci lint valid on the full config.
  • Replayed synthetic pipeline.failed payloads through applicable?: product-security/inventory and product-security-engineering/token-hunter-reports match, gitlab-org/gitlab does not.
  • Confirmed against gitlab.com's live GraphQL schema that aiAction takes chat: { resourceId, content } and returns requestId, read back via aiMessages(requestIds:, roles: [ASSISTANT]).
  • Confirmed RCA is available for these namespaces: troubleshootJobWithAi: true on a real failed pipeline (product-security/inventory pipeline 2804549805) and duoFeaturesEnabled: true on both a gitlab-com/gl-security and a gitlab-private/gl-security project.

I have not yet made a live /troubleshoot call, so the response text and latency are still unmeasured. I plan to validate that during rollout with the securitybot token, behind the unset-token switch.

Before this does anything

Nothing changes on merge until these are in place, which is deliberate:

  • a GitLab Duo seat for gitlab-securitybot, and the GITLAB_SECURITYBOT_API_TOKEN CI/CD variable plus Runway/Vault copy
  • pipeline_events webhooks on the in-scope projects pointing at the triage-ops endpoint
  • gitlab-securitybot added as Reporter on product-security-engineering/token-hunter-reports, the one in-scope project where it is not currently a member

Worth a reviewer's eye: these are security tooling pipelines, so job logs can contain sensitive values. RCA sends log segments to the AI Gateway and the summary lands in Slack and in triage-ops' logs.

Action items

  • Environment variable for a reactive processor added to config/triage-web.yaml and .gitlab/ci/triage-web.yml
  • Handbook page for Triage Operations — deferring to the reviewer on whether a Security-department processor belongs there; documented in doc/pipeline_failure_management.md for now
  • Affected groups:
    • Relevant Slack channels => #sec-prod-sec-eng-automation-failures

Merge request reports