Flaky test: TestDrainTrigger_ScopeDrainedAtTheNormalCadenceKeepsItsHash reads repo-scoped counters as [0 0] after a tick's chunks are enqueued
What fails
TestDrainTrigger_ScopeDrainedAtTheNormalCadenceKeepsItsHash/repo_scoped, in internal/accounting/drain_trigger_integration_test.go, fails non-deterministically on one Postgres version at a time.
drain_trigger_integration_test.go:1247, reached from :586 and :590
Error: Condition never satisfied
expected: [2]int64{3, 500}
actual : [2]int64{0, 0}
Messages: the repo scoped counters never reached [3 500] after the tick's chunks were enqueuedThe counters read [0 0] rather than a partial value, so the work does not appear to run at all inside the wait rather than running slowly.
Evidence that it is non-deterministic rather than a defect
First seen on merge request !1871 (merged), pipeline 2783561754, job 16055729311, on commit 4aed91dd.
| Check | Result |
|---|---|
test:integration: [POSTGRES, 18] |
failed |
test:integration: [POSTGRES, 17] and [POSTGRES, 16] |
both passed, on the same commit |
The same job on the previous pipeline 2783313393, commit 6cc6dac26 |
passed |
The target branch's own latest pipeline 2782905427 |
success |
Does !1871 (merged) touch internal/accounting? |
No. Its 23 changed files include none, and the test file last moved in a570c673f |
Two of three matrix entries running the same test on the same commit and passing is what makes this non-determinism rather than a code fault. The subtest ran 62.03s before giving up.
Relationship to issue #817 (closed)
Issue #817 (closed) tracks TestUploadEmit_ArtifactNewToRepositoryAndNamespace_MovesAllFourCounters, which reads namespace_statistics as 0 after a clean drain. These are different tests and neither should be closed as a duplicate of the other. They are recorded together because the shape is the same and a shared root cause is plausible but unestablished: both are internal/accounting integration tests, both wait for a drain to move a counter, and both read zero rather than a partial value. Issue #817 (closed) was seen on Postgres 17 and this one on Postgres 18, so the version is not the discriminator either.
Whoever investigates one should read the other first.
Condition and event: what a later reader will find
While no fix has landed, a red test:integration job naming this test on a branch that does not touch internal/accounting is this flake, and a retry of that single job is the response. Once a fix lands, the same red is a regression and needs the trace read rather than a retry.
The two counters the assertion names are the repository-scoped pair, so a fix that changes the drain's fan-out or its per-scope claim will move this test's timing and should re-run it deliberately rather than rely on the matrix.
What has not been established
- The root cause. Nothing here distinguishes a slow drain worker from a job that is never picked up.
- Whether it reproduces locally. It was not run locally against this branch: the file is behind
//go:build integration, and the run that cleared this branch's other obligation wasgo test -short ./..., which does not compile it. A cleaninternal/accountingline in that run is therefore not evidence about this test, and should not be read as any. - Frequency. One occurrence is recorded here.
This is a bot message