## Summary When an RSpec lane has a failed example, it reruns it in a new process after its main run. With dynamic test balancing, all the other lanes finish together, so the lanes that rerun failures finish about **2.5 min** after the rest in nearly every pipeline. The rerun takes 166s (p50), but the failing examples take only 5s of that. About 85s of each rerun goes to two steps that don't need to run in the retry process: 1. **About 38s:** the retry process rescans the whole job's `auto_explain` log, duplicating what the first process already wrote. 2. **About 48s:** after the reports are merged, most likely the working-tree `git diff` in `warn_on_successfully_retried_test`. Removing both would bring a rerun down to about 80s and roughly halve the tail. ## Problem Data from 96 tier 3 MR pipelines (2026-10-13 00:00 to 2026-10-14 06:00 UTC), with `rspec-ee unit pg17` at 47 lanes. I fetched the logs of every lane that finished more than 45s after the median lane (415 lanes). | | Value | |---|---| | 90th-percentile lane finish, after the median lane | 18s (p50 across pipelines) | | Last lane finish, after the median lane | 146s p25 / **156s p50** / 184s p90 | | Pipelines where the last lane reran failures in a new process | **91 / 96** | | Lanes rerunning failures per pipeline | 1-6 (284 in total). Every pipeline had at least one. | | When rerun lanes empty the queue, relative to the median lane's finish | 43s to 1s before it | | Pipelines where `rspec-ee unit pg17` is on the critical path | 29 / 96 | Rerun lanes finish their share of the queue on time. All of the extra time comes from the rerun. Here's the time from `Queue drained` to the end of the job script, across the 284 rerun lanes: | Step | p50 | p10-p90 | |---|---|---| | Exit the first RSpec process | 5s | 5-8s | | Boot the new process and load spec files | 67s | 58-74s | | Run the failing examples | **5s** | 2-9s (max 13s) | | After the examples, before `Finished in` (mostly `AutoExplain.record`) | 36s | 29-44s | | From `Saved rspec/rspec-<job>.json` to `A test was flaky...` (`junit_merge`, then `git diff` in `warn_on_successfully_retried_test`) | 49s | 40-54s | | **Total** | **166s** | 150-185s | <details> <summary>Example lane</summary> Pipeline [2945359575](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2945359575), lane 27/47 ([17160493593](https://gitlab.com/gitlab-org/gitlab/-/jobs/17160493593)): | Time | Event | |---|---| | 06:03:12 | `Queue drained. Exiting with status 1.` | | 06:03:18 | `Retrying the failing examples in a new RSpec process...` | | 06:04:29 | Retry process booted (`files took 1 minute 8.65 seconds to load`) | | 06:04:33-06:04:37 | The failing example group runs in 3.97s | | 06:05:07-06:05:15 | `auto_explain log contains 1090 entries for main` (took 30.1s), then `84` for `ci` (4.2s) and `156` for `sec` (4.2s). These are the same counts the first process wrote at 06:03:00-06:03:08. | | 06:05:16 | `Saved rspec/rspec-17160493593.json.` | | 06:06:06 | `A test was flaky and succeeded after being retried...` | </details> ## Proposal ### 1. Skip `AutoExplain.record` in the retry process [`AutoExplain.record`](https://gitlab.com/gitlab-org/gitlab/-/blob/96a7b1a80632303ff3228ff56f830c6667631e3d/spec/support/database/auto_explain.rb) runs in `after(:suite)`. It reads `log/pglog.csv`, the PostgreSQL service's log for the whole job, not just for the current process. So the retry process rescans the first process's 30 minutes of queries and writes another file with essentially the same entries. [`scripts/merge-auto-explain-logs`](https://gitlab.com/gitlab-org/gitlab/-/blob/96a7b1a80632303ff3228ff56f830c6667631e3d/scripts/merge-auto-explain-logs) then dedupes them by fingerprint. `retry_failed_rspec_examples` already exports `RSPEC_RETRY_PROCESS=true`, so: ```ruby def record_auto_explain?(connection) return false unless ENV['CI'] return false if ENV['RSPEC_RETRY_PROCESS'] == 'true' # ... ``` The only thing lost is queries that run only in the retry. The retry reruns examples that already ran in the first process, so their queries were already recorded, except for code paths that only run when an example passes. If those matter, an alternative is to keep `record` but only read rows logged after the retry process started (`log_time >= <process start>`). That still pays the roughly 4s scan per database, so it saves about 26s instead of 38s. ### 2. Make the "flaky test is part of this MR" check compare commits, not the working tree [`warn_on_successfully_retried_test`](https://gitlab.com/gitlab-org/gitlab/-/blob/96a7b1a80632303ff3228ff56f830c6667631e3d/scripts/rspec_helpers.sh#L382-404) starts with: ```shell local changed_files=$(git diff --name-only $CI_MERGE_REQUEST_TARGET_BRANCH_SHA | grep spec) ``` With only one commit given, `git diff` compares that commit with the **working tree**. RSpec jobs get their checkout from the `clone-gitlab-repo` artifact (`GIT_STRATEGY: none`), so the file metadata recorded in the index (modification times, inode numbers) no longer matches the files on disk. Git then probably has to read and hash many of the roughly 112k tracked files to rule out changes. `junit_merge` isn't the slow step. On this lane's real reports (1.1 MB, 3,311 test cases, one retried example) it takes about 0.5s locally. Proposed change: ```shell local changed_files=$(git diff --name-only "${CI_MERGE_REQUEST_TARGET_BRANCH_SHA}" HEAD | grep spec) ``` This compares two commits' trees and doesn't touch the working tree. It also better matches what the check means, "files changed by this MR". In merged-results pipelines, `HEAD` is the merge commit and the target SHA is its first parent, so both are in the depth-20 clone. The cause is inferred, not measured. First confirm it by timing the two steps separately, for example by wrapping `junit_merge` and the `git diff` in `time` in a draft MR. ### Remaining cost After both changes, a rerun is mostly the roughly 67s boot of the retry process and about 10s of exiting and reporting. Reducing the boot is a separate problem. ### Not pursued I considered changing the test balancer so that a lane with a failure stops claiming near the end of the queue, letting its rerun overlap the other lanes. It isn't worth it: - It's specific to the separate-process retry. - It needs coordination on the server. Otherwise, if every lane that's still claiming has a failure, they could all stop and leave tests unrun. ## How to verify - In lane logs with a rerun, there should be no `auto_explain log contains` lines after `Retrying the failing examples in a new RSpec process...`. - In the same logs, `Saved rspec/rspec-<job>.json` should be followed by `A test was flaky...` within a few seconds. - For `rspec-ee unit pg17`, the last lane should finish about 70s after the median lane (p50), down from about 156s. ## Related - gitlab-org/gitlab#633795: speed up `AutoExplain.record` in every RSpec process, not just the retry - #606465: dynamic test balancing - Found with [engwan/gitlab-pipeline-duration-agent#7](https://gitlab.com/engwan/gitlab-pipeline-duration-agent/-/work_items/7)