## Bug Description `TestCachingResolver_ConcurrentMissesCoalesce` in `internal/iam` failed one time on a developer machine. The same test passed on two later runs. This report records the sighting and the mechanism that makes the test sensitive to host load. It does not claim a defect in `CachingResolver`, the code the test covers. The section named `## Suggested Fix` gives two readings and picks neither. **The failure observation is second-hand.** Another run reported it, and this report did not see the failure. Every claim about the test and the code below was read from the source for this report, at the commit named beside it. Every claim about the failing run itself is marked as second-hand where it appears. The mechanism is named, and it is not an unexplained failure. The test releases 20 goroutines together, sleeps a fixed 50 ms, then releases the blocked leader call. That 50 ms is the entire margin for all 20 goroutines to reach the `singleflight` entry. The test's own comment states what happens to a goroutine that does not arrive in time, and that comment is quoted below. ## Steps to Reproduce No reliable reproduction exists. Reproducibility: observed one time, second-hand, against two later passes reported by the same run. Reproduced independently: **no.** Two bounded attempts ran for this report. Both passed. | Command | Runs | Result | Wall time | | --- | --- | --- | --- | | `go test -race -count=20 -run 'TestCachingResolver_ConcurrentMissesCoalesce' ./internal/iam/` | 20 | pass | 2.060 s | | `go test -short -count=20 -run 'TestCachingResolver_ConcurrentMissesCoalesce' ./internal/iam/` | 20 | pass | 1.016 s | The second command mirrors the flags of the `go-test` pre-commit hook, which is the reported condition. That hook comes from `gitlab-com/gl-infra/common-ci-tasks` at `v5.2`, and its script `scripts/pre-commit/go-test.sh` runs one command: ```shell go test -short ./... ``` `internal/iam` contains no call to `testing.Short()`, from `grep -rn 'testing.Short()' internal/iam/`, which returns nothing. This test therefore runs under that hook. The hook runs the whole repository, so every package binary competes for the same cores. Both attempts ran at `4ad027eee48a1d965f0bd45bd33cbbbf1511ec12`. `internal/iam` at that commit is identical to `origin/main` at `094c046a778698bac3003dc1fb0d17006c350c0a`: `git diff --stat origin/main HEAD -- internal/iam/` prints nothing. The environment carried `GOFLAGS=-tags=integration,development_stubs`. ## Expected Behavior The test passes on every run. ## Actual Behavior The test failed one time. **The failure output is not recorded in this report.** The relay that carried the observation gave no `--- FAIL` block, no assertion message, and no duration. It gave three further facts, and none of them is verified here: 1. The failure came from a pre-commit hook run. 2. The host was running a large parallel agent fleet at the time. 3. The branch that carried the run changes no file under `internal/iam`. Fact 3 **is not checkable from this report**, because the relay named no branch and no commit. Issue [#1056](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1056) carries the same claim for a different test and names its branch and merge request, so that one was checkable. This one is not. Treat all three as second-hand. ### One correction to the relayed report The relay put the sleep at `internal/iam/caching_internal_test.go:344`. That line is blank. The real coordinates, read at `4ad027eee` and identical at `origin/main` `094c046a7`: | What | Line | | --- | --- | | `func TestCachingResolver_ConcurrentMissesCoalesce` | 323 | | `t.Parallel()` | 324 | | `const n = 20` | 332 | | `time.Sleep(50 * time.Millisecond)` | 372 | | `require.Equal(t, int64(1), inner.calls.Load(), ...)` | 377 | ### The test documents its own failure mode This is the strongest evidence in the report, and it is in the test itself. Quoted verbatim from `internal/iam/caching_internal_test.go`, lines 365 to 372: ```go // internal/iam/caching_internal_test.go — TestCachingResolver_ConcurrentMissesCoalesce // singleflight.Group has no externally observable "N callers have // joined this call" signal — only the leader's own call is visible via // inner.calls. This bridges the gap between "released from the gate" // and "actually inside sf.Do" (a mutex lock and a map lookup) before // the blocked leader call is allowed to return; without it, a follower // still short of that entry when unblock closes starts a second, // independent call instead of joining the first. time.Sleep(50 * time.Millisecond) ``` A second independent call makes `inner.calls.Load()` return 2. The assertion at line 377 wants 1. One detail in that comment is stale, and it does not change the mechanism. The comment names `sf.Do`. The production path calls `c.sf.DoChan` at `internal/iam/caching.go:110`. Both take the group mutex and look the key up in the in-flight map, so the window the comment describes is the same window. ### What the 50 ms has to cover The test gives the sleep two separate jobs, and no signal separates them: 1. The leader goroutine must reach `DoChan` and register the key in the in-flight map. 2. The other 19 goroutines must reach `DoChan` and find that key already registered. Job 1 is observable in the fixture and is not used. `blockingResolver.ReadRelationships` runs `b.calls.Add(1)` before it blocks on `<-b.unblock`, so the leader announces its own arrival, and the test ignores that announcement. Job 2 is not observable. `singleflight.Group` publishes no waiter count, so no signal can replace the sleep for this half. The test also runs under `t.Parallel()`, at line 324. The per-file counts from `grep -rc 't.Parallel()' internal/iam/` sum to 51 across the package's test files. Two of the 51 are in `client_integration_test.go`, which compiles only under the `integration` build tag. ### A short sleep can also weaken the test without failing it This is read from the code, not observed. `store` runs inside the `DoChan` closure, at `internal/iam/caching.go:141`, before that closure returns. `load` returns `slices.Clone(entry.relationships)`, at `internal/iam/caching.go:180`. A straggler that arrives after the leader's closure returns therefore takes the cache-hit path. It gets an independent copy, it never reaches the inner resolver, and `inner.calls` stays at 1. Every assertion passes, and that straggler proved nothing about coalescing. So the same shortfall reads as a failure in one ordering and as a silent pass in another. ## Suggested Fix **This section gives two readings. Neither is established, and one failure cannot separate them.** ### Reading 1: the loaded host is the proximate cause, and the test is innocent A machine running a large parallel fleet starves goroutines of scheduler time. Under `go test -short ./...` the whole repository competes for the same cores. A 50 ms budget for 20 goroutines is comfortable on an idle host and thin on a saturated one. This reading explains one failure against two passes better than a product defect does. ### Reading 2: the fixed sleep is why load can cause a failure at all This is a statement about the test's design, not about the failing run. The margin is a constant. The work it has to cover scales with the goroutine count and with the load on the host. Nothing in the test measures either. A design of this shape degrades as the host gets busier, and it degrades without any change to `internal/iam`. The two readings are compatible. Reading 1 names the trigger. Reading 2 names why the trigger has an effect. ### An in-repo precedent for the shape of a fix `internal/format/npm` solved the same problem for the same library. `TestPackument_SingleflightInlineBuild`, at `internal/format/npm/packument_get_test.go:1567`, splits the wait into three parts: 1. A test-only seam, `PackumentHandler.SetBeforeSingleflightDo` at `internal/format/npm/export_test.go:89`, parks every request immediately before `builds.Do`. The test releases all of them together. 2. A barrier signals when the leader's fill has entered and blocked. This removes job 1 above from the sleep entirely. 3. Only then does a bounded sleep cover the join of the remaining waiters. That sleep is 100 ms and is documented as the residue that no signal can replace. That test also logs the observed count when it is not 1, so a failure names the number instead of only the expectation. The npm test still sleeps, and its comment says why: ```go // internal/format/npm/packument_get_test.go — TestPackument_SingleflightInlineBuild // singleflight exposes no waiter count, so that post-entered settle cannot // be replaced by a fully deterministic "all N-1 registered" signal — an // inherent limitation of deterministically testing a singleflight collapse. ``` So a full removal of the sleep is not available. Three options are, and each is independent of the others: - Add the leader-entered barrier to `blockingResolver`, so the sleep covers the waiter join only. - Add a `t.Logf` on a count other than 1, so a second failure names the observed number. - Raise the 50 ms, which buys margin and changes nothing structural. ### What a second sighting must capture 1. The whole `--- FAIL` block, with the file, the line, and the observed count. 2. The host load average and the number of concurrent test processes at that moment. 3. The branch and the commit under test. 4. Post all three in a comment on this issue. ## Suggested DRI Workstream S09 AuthZ, DRI @bmarjanovic, from the `### Workstream assignments` table in `docs/roadmap/closed-beta.md`. `internal/iam` is the relationship-lookup client that the authorization path calls, and it is not a format package. The test and the code under it arrived in [!876](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/876), `feat(iam): cache read-only relationship lookups (#219)`, authored by @bmarjanovic and merged on 2026-08-07. This is a suggestion for triage, not an assignment. The issue is unassigned. ## Labels to Apply - `type::maintenance` — a flaky test is upkeep, not a defect in shipped code. The three most recent flake issues on this project carry the same type: [#1056](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1056), [#1034](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1034) and [#1014](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1014). - `maintenance::pipelines` - `Category:Artifact Registry` - `artifact-registry::authz` — the stream that owns `internal/iam`, matching [#456](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/456) and [#455](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/455) - `backend` - `golang` No `severity::` label, because `docs/dev/labels.md` requires that scope for `type::bug` only. No `format::*` and no `repo-type::*` label. Issue [#1056](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1056) carries both because its test is in `internal/format/oci`. `internal/iam` is not a format package and serves no repository type, so neither axis applies. ## Additional Context | | | |---|---| | Environment | Local developer machine, Linux, 22 cores, 62 GB RAM | | Org slug | _Not applicable — no service was running._ | | Repository | _Not applicable — this is a Go unit test._ | | Format | _Not applicable — `internal/iam` is not a format package._ | | Auth method | _Not applicable._ | | Client and version | `go test`, Go 1.26.7, against `go 1.26.0` in `go.mod` | | First observed (UTC) | Not recorded. The relay gave no timestamp. | | Correlation ID | _Not applicable._ | | Workaround | Re-run the test. The relay reports two later passes. | ### Load on the host The two reproduction attempts in this report ran on a 22-core developer machine that was itself under fleet load. `uptime` read a 1-minute load average of 11.47 before the first attempt and 15.70 after it. Those figures describe this report's own attempts. Whether that host is the host that saw the failure is second-hand and not verified here. This report has no measurement of the load at the moment of the failure. ### Duplicate search Every search below ran against `gitlab-org/ops/artifact-registry` on 2026-09-02. Each empty or negative result has a positive control on the same tool, so an empty list means absence and not a broken query. | Search | Tool | Result | | --- | --- | --- | | `caching_internal_test` | `glab issue list --search ... --all` | 1 hit, [#1014](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1014), a tokenized match on "cache". Its body names no `iam`, no `caching_internal`, and no `singleflight`. Not a duplicate. | | `ConcurrentMissesCoalesce` | `glab issue list --search ... --all` | 0 hits | | `CachingResolver` | `glab issue list --search ... --all` | 2 hits, [#861](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/861) and [#527](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/527), neither a flake. This is the positive control for the row above. | | `singleflight` | `glab issue list --search ... --all` | 18 hits, none a flake in `internal/iam` | | `internal/iam` | `glab issue list --search ... --all` | 6 hits, none a flake. The nearest is [#527](https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/527), which tracks observability and partial eviction for the same cache's full-clear path. Different code path, no flake. | | `flaky` | `glab issue list --search ... --all` | 19 hits, none in `internal/iam` | | `flake` | `glab issue list --search ... --all` | 14 hits, none in `internal/iam` | | `ConcurrentMissesCoalesce` | project search, `scope=notes` | Hits only on [!876](https://gitlab.com/gitlab-org/ops/artifact-registry/-/merge_requests/876), the merge request that added the test. Every hit is a review discussion from that merge request, and none reports a test failure. | | `ConcurrentMissesCoalesce` | `glab mr list --search` | 0 open merge requests | | `iam` | `glab mr list --search` | 5 open merge requests. This is the positive control for the row above. | The issue searches read titles and bodies. The notes search covers comments, which is the gap a title-and-body search leaves. ### What this report did not do - It did not see the failure. The observation is second-hand. - It did not obtain the failure output. No `--- FAIL` block was captured. - It did not check the branch that carried the failing run. The relay named no branch and no commit. - It did not run the whole package in a loop. The two bounded attempts above were the whole reproduction budget, because the host is under load and a loop on it proves nothing either way. - It did not establish a defect in `CachingResolver`.