FFP flow occasionally freezes and have to be manually restarted

Users have reported that they kickoff the flow and then the flow just hangs.

1. Intermittent hangs during feature invocation represent a recurring friction point throughout the day. Out of approximately 20 pipeline debug sessions in a single day, 3–4 required a reload due to the feature hanging or failing to respond. (Call recording)

Internal Data

  1. FFP Dashboard
  2. Duo Workflow Dashboard

Objective

We need to have our reporting corrected so that we can verify that at least 98% of the flows that are started and not aborted by the users - reach completion.

Proposal

  1. Verify what our current completion rate is for last 4 weeks
  2. Propose steps necessary to improve the completion rate if it's less than 98%

Progress

Reporting fix (this milestone)

  • Post diagnosis and follow-up proposals as a comment
  • Coordinate with @halilcoban on clean_stuck_workflows_service.rb
  • Create MR: emit agent_platform_session_dropped when a stuck session is cleaned up: !249723 (merged)
  • Ask the data team to add the stuck bucket to the FFP Tableau charts
  • Get the 4-week completion rate from Snowflake (@halilcoban)

Reducing the stall window

  • @tigerwnz called it: a green pipeline next to a still-running session counts as a failure (note)
  • Create MR: drop the session when its workload pipeline finishes cleanly but the session is still created/running, instead of waiting up to 60 minutes for the cleanup cron: !250072 (merged)

Confirming the stall

  • Determine whether duo CLI retry exhaustion exits non-zero or clean (needs an executor job log from a stalled session). The client already has heartbeat, keepalive and 5-attempt backoff, so the earlier "add reconnect handling" framing was wrong
  • Warn on FF rollout #607029 that duo_workflow_write_incremental_only breaks stuck detection
  • Optional follow-up, pending verification: ee/lib/gitlab/duo_workflow/sandbox.rb line 77 ends every workload script with echo "Command execution completed with exit code: $?", which prints the exit code but discards it. Whether that actually masks a failing job depends on whether the runner applies set -e, so it needs the same job log

Triage

  • Confirm the group label (the code involved is group::agent execution, not group::pipeline execution)
  • Split reporting versus stall into separate issues
  • Set 19.3 expectations: the reporting fix does not stop the hangs

Decided, no action

  • No historical backfill: cleaned-up sessions never emitted event 20 and there is no source to replay from
Edited by Peter Gumeson