Add idempotency/locking for SPEP test run pipeline association

What does this MR do and why?

Addresses the non-atomic write issue between CI database (pipelines) and main database (test runs) when creating SPEP test run pipelines.

Related to #591425 (closed)

Problem

When creating a test run, the system writes to two separate databases without atomic transactions:

  1. CI database — pipeline record
  2. Main database — test run record with pipeline association

This can result in:

  • Orphaned pipelines if association fails after pipeline creation
  • Duplicate pipelines from worker retries
  • Test runs stuck in pending state

Solution

Add a new creating state and use an atomic state transition to claim the test run before pipeline creation:

UPDATE ... SET state = 'creating' WHERE id = ? AND state = 'pending'

This ensures only one worker can proceed with pipeline creation. If the atomic update returns 0 rows, another worker already claimed it.

Changes:

  • Add creating state to ScheduledPipelineExecutionPolicyTestRun model
  • Add claim_for_pipeline_creation method for atomic claiming
  • Add mark_as_failed! method (reused in TestRunCompletionWorker)
  • Update worker to use atomic state transition instead of exclusive lease locking

If pipeline creation fails after claiming, the test run is marked as failed with an error message indicating an orphaned pipeline may exist. The user can then start a new test run.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Andy Schoenen

Merge request reports

Loading
Loading