Implement bulk insert for pipeline records

Laura Montemayorrequested to merge
implement-bulk-insert-pipeline-records into master

What does this MR do and why?

Implements bulk insert for pipeline records to reduce database round trips during pipeline creation.

Records bulk inserted:

  • Stages
  • Builds (CommitStatus)
  • Execution configs
  • Job definition instances
  • Build needs
  • Job sources

Changes:

  • Saves pipeline first, then bulk inserts child records in batches of 500
  • Uses bulk_insert! with returns to restore IDs after insert
  • Handles IID conflicts by flushing internal IDs and returning error
  • Cleans up pipeline on failure
  • Behind ci_bulk_insert_pipeline_records feature flag

References

Closes https://gitlab.com/gitlab-org/gitlab/-/issues/588748

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Laura Montemayor

Merge request reports