Container remote token cache is not evicted on a credential-only settings change

The settings PATCH route (feat(managementapi): serve the remote settings ... (!1732 - merged) • Hayley Swimelar • 19.4) updates a container remote's credentials without evicting that repository's cached upstream tokens. A token minted under the old credentials keeps serving until its natural expiry, so a credential rotation or revocation does not take effect immediately on the container path.

oci.RemoteTokenCache.EvictRemote carries the contract that a token minted from replaced upstream credentials must not outlive them. Nothing performs that eviction today, on either arm.

ContainerRemoteRepositoryStore.UpdateContainerRemoteRepository names three caller-side companions that are not the store's work: the per-process token-cache eviction, the eviction enqueue that rides the transaction, and the failure-counter reset after commit. The enqueue and the counter reset ride a url change only. The token-cache eviction has to ride a url change and a credential replace or clear, because replacing a credential without re-pointing the upstream leaves a token minted from the old secret usable until it expires. See When it becomes a live defect for how the url half may be closed without eviction at all.

Scope

Eviction of the remote token cache keyed by the repository whose credentials changed. Maven and npm resolve credentials per request and do not carry this window.

When it becomes a live defect

The obligation falls due on whichever change wires a token cache into the container read path. !1717 (merged) is that change: it adds the tree's only production oci.NewRemoteTokenCache construction, in buildOCIUpstreamAuthenticator. While !1717 (merged) is open the container read path reaches no cache, so no arm has a caller to get wrong; once it merges, a credential replace or clear leaves a token minted from the old secret usable for its remaining lifetime.

!1717 (merged) also carries a fix for the url arm that needs no eviction: the upstream URL joins RemoteTokenKey, so a url change misses the cache by construction rather than by a caller remembering to evict. If it merges in that shape this issue narrows to the credential arm alone; if the key change is dropped, the url arm is in scope here too.

Wiring eviction on the url arm alone would look complete and leave credential rotation silently ineffective for the token's remaining lifetime.

What a solution has to account for

EvictRemote is a partial fix for this arm, not a complete one. Its doc comment says "Other instances self-heal through the retry-on-401 path, because the cache is per-process." That holds for a url change, where the new upstream rejects a token it did not mint. It does not hold here: the url is unchanged, so the upstream still accepts the token minted from the old credentials and answers 200, and no 401 ever fires. Eviction therefore fixes the one instance that processed the write and leaves every other instance serving the stale token until it expires. A solution that stops at wiring EvictRemote should say so rather than read as complete.

Bounding token_cache_max_ttl is the one lever that works across instances today. The stored lifetime is min(response.expires_in, token_cache_max_ttl) - token_cache_expiry_margin. The default ceiling is 1h (config.example.yaml:198, docs/dev/configuration-reference.md:589). Most registries issue expires_in: 300, so the ceiling rarely binds, which is why lowering it costs little and caps the case with the largest exposure: an upstream that issues a long-lived token. It bounds the window rather than closing it, so it is a mitigation and not this issue's fix. Changing the default also pulls in config.example.yaml and docs/dev/configuration-reference.md, per the guardrail requiring the configuration reference to move with configuration code.

Where the obligation is currently recorded

Only in a doc comment on UpdateContainerRemoteRepository, in internal/datastore/container_remote_repositories_update.go. That comment names oci.RemoteTokenCache.EvictRemote in prose rather than as a doc link, because internal/datastore does not import internal/format/oci. A comment is the wrong home for an obligation that falls due in a different change, which is what this issue is for.

The credential-only arm became reachable with feat(managementapi): serve the remote settings ... (!1732 - merged) • Hayley Swimelar • 19.4, which serves the remote settings PATCH and performs credential replace and clear without a url change.

Surfaced by the step-23 build review on feat(managementapi): serve the remote settings ... (!1732 - merged) • Hayley Swimelar • 19.4. Content merged from #717 (closed), closed as a duplicate.

Edited by Sylvia Shen