feat(renovate): audit that Renovate is delivering, fleet-wide
Configuration conformance was standing in for delivery, and the two came apart for three months.
kaniko shipped a well-formed renovate.json extending the shared preset, passed every assertion in the reference's renovate standard, and received nothing. Three push rules (reject_unsigned_commits, commit_committer_check, member_check) rejected the bot, so no branch was ever created. Its Dependency Dashboard said Error updating branch continuously and nothing in the estate read it. It is the only project of 58 carrying any of those rules.
This adds a job in the renovate stage, running after the Renovate job in the same scheduled pipeline, that reads every enrolled project's dashboard and fails when Renovate could not deliver.
Why fleet-wide and not a per-project standard
| Reason | Detail |
|---|---|
| Token scope | Reading projects/:id/push_rule needs Maintainer. A per-project version means distributing a privileged token across 58 projects, a worse posture than the thing being checked. |
| Absence is invisible from inside | "Renovate never visited me" cannot be self-detected. An unscanned project looks identical to one with no dependencies. That distinction only exists against the enrolled set. |
| No per-project delivery exists | reference-check exists at no catalog tag, so a standard written there would run nowhere. |
The bot token already has exactly the access needed, because it is the token that writes the dashboards being read. No new credential.
It blocks only on delivery failure
The first run proved this distinction matters. Fails on a write-side problem (Error updating branch and similar) or updates stuck in the dashboard's Errored section.
Reports and never fails on a problem where nothing is stuck: lab/airgap-fips reports Error obtaining docker token for registry.airgap-fips.local, a registry unreachable on purpose, on files under bench/ that the shared preset already freezes as evidence. A first cut blocked on that too, which would have held a lab permanently red for doing its job. The write-side/lookup distinction is drawn from those two measured instances, not from guesswork.
Freshness is deliberately not asserted. "No update in 90 days" is uncheckable, because a project whose dependencies are current looks exactly like one that is broken. Inferring health from silence is how kaniko went unnoticed, so the audit reads a signal that states itself.
Verification
--selftest runs the classifier against five fixtures trimmed from real dashboards, with no network, and is the job's first script line.
One fixture exists purely to isolate the write-side marker. Without it, breaking WRITE_SIDE_MARKERS left the suite green, because every other failing fixture also had stuck updates and errored > 0 decided them. Verified by breaking the marker and watching only that case go red.
First run against the live estate: 58 enrolled, 44 healthy, 0 failing to deliver, 1 non-blocking, 13 with no dashboard.
Paired change
The reference's renovate standard is corrected in reference!62: it stops requiring a renovate.json to exist (enrollment is inherited now), teaches the canonical preset reference instead of the deprecated shim path, and names this job in its unasserted: field as what audits the property it cannot.