Loading
Tell a shed request apart from a deleted file in drift-check
What this does
drift-check exists to notice that an upstream symbol has gone, and the skill
tells an operator that a FAIL here is real drift to chase. Until now any
non-200 response produced an empty body, and assert read an empty body as a
deleted file. So the two ways gitlab.com says "ask again" were reported as
deletions: 503 when Gitaly sheds load, 429 when it rate-limits.
A run this morning reported five failures. All five files were live. One was
ee/lib/gitlab/duo/administration/verify_self_hosted_setup.rb, whose absence
would have read as upstream removing the tool the docs send administrators to.
A second run minutes later disagreed with itself, three failures instead of
five, which is the clearest sign the result was measuring the API's mood rather
than the source.
fetch now retries the retryable statuses and reports three outcomes instead
of two: served, absent on a 404, or unreachable. Only a 404 counts as drift. An
unreachable assert gets its own bucket, and the summary says plainly that
nothing in the register needs touching for it, since the failure worth avoiding
is an operator chasing a change that never happened.
The cost is that a broad outage makes a run slow rather than fast and wrong. For a scheduled canary that is the right way round.
Verified
Against a local server returning 503 four times and then 200, so all three outcomes were observed rather than assumed:
- the retry case passes
- a genuine 404 still reports the file gone
- a refused connection reports unreachable
Against real master the check now reports 69 passed and 0 failed, where the same run before this change reported 49 passed and 5 failed.
FETCH_RETRIES and FETCH_BACKOFF are overridable, in case five attempts with
sleeps of 3, 6, 9 and 12 seconds is the wrong shape for CI.