Make the RSpec retry process cheaper

Heinrich Lee Yurequested to merge
633791-cheaper-rspec-retry into master

What does this MR do?

Makes the RSpec retry process (RETRY_FAILED_TESTS_IN_NEW_PROCESS) about 85s cheaper. With dynamic test balancing, the lanes that retry a failure finish about 2.5 min after the rest of the group in nearly every pipeline. The failing examples take only about 5s of that. This MR removes the two biggest avoidable steps:

  1. Empty the auto_explain log after recording it (about 38s). AutoExplain.record reads log/pglog.csv, the PostgreSQL service's log for the whole job. So the retry process rescanned everything the first process had already recorded, and wrote a second file with essentially the same entries. In the example lane below, both processes wrote 1090 / 84 / 156 entries for main / ci / sec. scripts/merge-auto-explain-logs then deduped them by fingerprint.

    Now, once every database is recorded, record empties the log with COPY (SELECT 1 WHERE false) TO '<data_directory>/log/pglog.csv'. That needs superuser, which record_auto_explain? already requires. The next RSpec process in the job then only scans and records its own queries. Nothing is lost, including queries that only a passing retry reaches.

    Tested against a throwaway PostgreSQL 17 started with the CI logging flags (logging_collector, csvlog, log_filename=pglog, auto_explain.log_min_duration=0, JSON format):

    • After truncation, PostgreSQL keeps appending from the start of the file. It writes no NUL bytes, and file_fdw still parses the file.
    • Running AutoExplain.setup/record through Rails twice, as a first process and then as a retry: the retry's files only contained the retry's own queries.

  2. Compare two commits in warn_on_successfully_retried_test (about 48s). git diff --name-only $CI_MERGE_REQUEST_TARGET_BRANCH_SHA with only one commit compares that commit with the working tree. RSpec jobs restore the checkout from the clone-gitlab-repo artifact, so the file metadata in the index no longer matches the files on disk, and git rehashes every tracked file. Comparing the target branch SHA with HEAD (the merged-results commit) only compares trees, and better matches what the check means: files changed by this MR. The old working-tree diff caught two extra things, and neither mattered:

    • In FOSS_ONLY jobs, it listed every ee/ spec file that the job's before_script deletes. Those could never match, because FOSS jobs only retry spec/ files.
    • Uncommitted edits, of which CI has none.

    In merge trains, HEAD is the train commit and the target SHA is the previous train car, so the result is still the MR's changes. Without a target SHA (pipelines without merged results), the diff is empty, as it effectively was before.

Locally, after making the stat data stale (git ls-files -z | xargs -0 touch -c), git diff --name-only <sha> took 29.4s and git diff --name-only <sha> HEAD took 0.04s.

Time from Queue drained to the end of the job script, across 284 rerun lanes of rspec-ee unit pg17 in 96 tier 3 MR pipelines (2026-10-13/14):

Step p50 p10-p90 After this MR
Exit the first RSpec process 5s 5-8s unchanged
Boot the retry process and load spec files 67s 58-74s unchanged
Run the failing examples 5s 2-9s unchanged
After the examples, mostly AutoExplain.record 36s 29-44s a few seconds (only the retry's own queries)
junit_merge (about 0.5s) and the git diff 49s 40-54s about 1s
Total 166s 150-185s about 80s
Example lane

Lane 27/47 of rspec-ee unit pg17, https://gitlab.com/gitlab-org/gitlab/-/jobs/17160493593:

Time Event
06:03:00-06:03:08 First process: auto_explain log contains 1090 entries for main (30s), then 84 for ci and 156 for sec
06:03:18 Retrying the failing examples in a new RSpec process...
06:04:33-06:04:37 The failing example group runs in 3.97s
06:05:07-06:05:15 Retry process: the same 1090 / 84 / 156 entries again (38s)
06:05:16 Saved rspec/rspec-17160493593.json.
06:06:06 A test was flaky and succeeded after being retried... (50s later)

Related issues

References

  • #633795 (closed): separate MR to make AutoExplain.record faster in every process
  • #606465: dynamic test balancing, which made this tail visible

Checklist

Pre-merge

Consider the effect of the changes in this merge request on the following:

  • Different pipeline types. The time saving only applies to RSpec jobs that retry a failure, or that run more than one RSpec process. Every job that records auto_explain now ends with an empty pglog.csv, and nothing else reads that file. In merged-results pipelines, the flaky-test check compares the merge commit with the target branch. Without CI_MERGE_REQUEST_TARGET_BRANCH_SHA, it falls back to an empty diff, as before.
  • Non-canonical projects:
    • gitlab-foss: the same scripts and record_auto_explain? conditions apply.
    • security: no difference.
    • dev: no difference.
    • personal forks: no difference.
  • Pipeline performance: saves about 85s per retrying lane, which shortens the tail of balanced RSpec groups by about the same.

If new jobs are added: not applicable, no new jobs.

This will help keep track of expected pipeline time and cost increases.

Post-merge

Edited by Heinrich Lee Yu

Merge request reports