> Filed by the `fuzz-run` finder (agentic fuzz-testing of AR). Labels are **proposals** — please confirm severity/subtype at triage, then flip `fuzz::triage` → `fuzz::approved` or close. `fingerprint: oci:mgmt-repo-DELETE-409-after-data-plane-empty-vs-GET-artifacts_count-0` ## Summary An OCI hosted repository that has **ever held an image cannot be deleted** through the documented management-API path. After fully emptying the repo via the OCI data plane (untag → delete manifest by digest → delete both blobs — each confirmed: `tags/list` → `[]`, manifest-by-digest → 404, both blob HEADs → 404), `DELETE /api/v1/<ns>/repositories/<repo>` still returns: ``` 409 {"code":"conflict","message":"repository still contains artifacts and must be emptied first"} ``` This persists indefinitely (verified unchanged after >5 min). Worse, it is **self-contradictory**: the same management API's `GET /api/v1/<ns>/repositories/<repo>` reports `artifacts_count: 0` for that repo — so AR simultaneously reports the repo empty (GET) and non-empty (DELETE precondition). The 409 is user-unactionable: it instructs the operator to empty a repo that is already empty by every signal AR exposes. A repo that was **never** populated in the same namespace deletes cleanly (204), isolating the wedge to the push-then-data-plane-empty lifecycle. ## Oracle & spec - **Oracle:** state-inconsistency (control-plane self-contradiction: `GET artifacts_count=0` vs `DELETE` "still contains artifacts"). - **Novelty:** **worse-than-documented.** `docs/specs/S17-rest-management-api.md:164` presents "empty via the format client APIs, then delete" as the Phase-1 workflow — but for OCI there is no client-API operation that removes the underlying row (see root cause), so the documented empty-then-delete path is **impossible** for any OCI repo that ever held an image. `docs/specs/S12-container-oci-hosted.md:608` only acknowledges orphan `container_images` rows *from abandoned sessions* (POST with no PATCH/PUT), reaped by ADR-011 async reconciliation; it does not cover a populated-then-emptied image, and reconciliation did not fire within the observed window. - The OCI data-plane deletes themselves are spec-compliant (202 → 404); the defect is solely in AR's control plane. ## Root cause (traced in source) - On push, OCI auto-creates a `container_images` row (the image name) via `INSERT ... ON CONFLICT DO NOTHING`. The data-plane deletes remove `container_tags`, `container_manifests`, `container_manifest_relationships`, and the blob-attachment link — but **never the `container_images` row**. There is no `DELETE FROM container_images` anywhere in `internal/` (the manifest-delete cascade in `internal/format/oci/manifest_delete.go` + `internal/datastore/container_manifest_deleter.go` deliberately excludes it). - The management `DELETE` issues `DELETE FROM repositories`. Child→`repositories` FKs are `ON DELETE CASCADE`, but the `container_images → container_repositories` FK is `NO ACTION`, so the surviving empty image row rejects the cascade with SQLSTATE `23503`, which `internal/datastore/repositories.go:mapRepositoryDeleteError` (766–819) maps to `ErrRepositoryNotEmpty` → `internal/managementapi/delete.go:writeDeleteStoreError` → 409. - **Why GET and DELETE disagree:** `DELETE` evaluates live artifact rows through the FK; `GET`'s `artifacts_count` (`internal/managementapi/resource.go:130`) reads a denormalized counter column on `repositories` that the **OCI write path never increments** (increments exist only in npm `publish_commit.go`/`unpublish_package.go` and maven `store.go`). The two endpoints answer from entirely different subsystems — a stat/count subsystem disconnected from the actual artifact rows. - **Fix direction:** the OCI data-plane delete path (or the repo-delete cascade) must remove the `container_images` row once its last manifest/tag is gone, and/or the delete precondition and the displayed counter should read the same source of truth. Related (not duplicates): #197 (S17 soft-delete model, feature) and #44 (`repositories.size_bytes` CHECK — same counter subsystem). ## Reproduction Fresh namespace from clean state (parameterized — no hardcoded slug): ```sh # from the fuzz-lab workspace on the e2e-fuzzy-tester VM eval "$(tooling/run-setup.sh repro-repodel oci)" export AR_TOKEN=$(sudo cat /srv/ar/bootstrap-token.txt) bash journal/1864315411-7/repo-delete-stuck-after-empty.repro.sh # pushes an image, empties it via the data plane (tags [], manifest 404, blobs 404), # then DELETE repo -> 409 "still contains artifacts"; retries after 90s -> still 409 ``` <details><summary>repo-delete-stuck-after-empty.repro.sh</summary> ```sh #!/usr/bin/env bash # Candidate: management-API repository DELETE permanently refuses with 409 # "repository still contains artifacts and must be emptied first" even after # every OCI-data-plane-observable signal (tags/list, manifest-by-digest HEAD, # both blob HEADs) confirms the repository is completely empty - and it does # not resolve after a 90s wait (ruling out simple async GC lag). # # Assumes the caller has already run, for SOME namespace key <K>: # eval "$(tooling/run-setup.sh <K> oci)" # export AR_TOKEN=$(sudo cat /srv/ar/bootstrap-token.txt) # i.e. AR_SLUG, AR_HOST, AR_OCI_PREFIX, DOCKER_CONFIG, AR_TOKEN are exported. # Never hardcodes a namespace slug - everything is parameterized on $AR_SLUG / # $AR_OCI_PREFIX so this replays in any fresh namespace. set -uo pipefail : "${AR_SLUG:?run tooling/run-setup.sh <key> oci and eval its output first}" : "${AR_HOST:?run tooling/run-setup.sh <key> oci and eval its output first}" : "${AR_OCI_PREFIX:?run tooling/run-setup.sh <key> oci and eval its output first}" : "${AR_TOKEN:?export AR_TOKEN=\$(sudo cat /srv/ar/bootstrap-token.txt) first}" command -v regctl >/dev/null || { echo "regctl not found in PATH" >&2; exit 1; } regctl registry set --tls disabled "$AR_HOST" >/dev/null 2>&1 REPO="oci-orphan-test-repro" IMG="${AR_HOST}/${AR_SLUG}/container/${REPO}/probe" api() { curl -s -w '\nHTTP_STATUS:%{http_code}\n' -H "Authorization: Bearer $AR_TOKEN" "$@"; } apicode() { curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $AR_TOKEN" "$@"; } echo "== 1. create fresh hosted oci repo '$REPO' in namespace $AR_SLUG ==" create_code=$(apicode -X POST "http://${AR_HOST}/api/v1/${AR_SLUG}/repositories" \ -H 'Content-Type: application/json' \ -d "{\"name\":\"${REPO}\",\"format\":\"oci\",\"visibility\":\"public\"}") echo "create -> $create_code" [[ "$create_code" == "201" || "$create_code" == "409" ]] || { echo "FAIL: unexpected create status $create_code"; exit 1; } WORK=$(mktemp -d) trap 'rm -rf "$WORK"' EXIT cd "$WORK" echo '{}' > config.json CONFIG_DIGEST="sha256:$(sha256sum config.json | cut -d' ' -f1)" CONFIG_SIZE=$(stat -c%s config.json) printf 'repro-layer-data\n' > layer.bin tar -cf layer.tar layer.bin LAYER_DIGEST="sha256:$(sha256sum layer.tar | cut -d' ' -f1)" LAYER_SIZE=$(stat -c%s layer.tar) echo "== 2. push config + layer blobs, then a manifest tagged 'v1' ==" regctl blob put "$IMG" --digest "$CONFIG_DIGEST" < config.json >/dev/null regctl blob put "$IMG" --digest "$LAYER_DIGEST" < layer.tar >/dev/null cat > manifest.json <<EOF {"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","digest":"$CONFIG_DIGEST","size":$CONFIG_SIZE},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"$LAYER_DIGEST","size":$LAYER_SIZE}]} EOF regctl manifest put "$IMG:v1" --content-type application/vnd.oci.image.manifest.v1+json < manifest.json >/dev/null DIGEST=$(regctl manifest get "$IMG:v1" --format '{{.GetDescriptor.Digest}}') echo "manifest digest: $DIGEST" NAME="${AR_SLUG}/container/${REPO}/probe" echo "== 3. delete tag 'v1' (untag only) ==" apicode -X DELETE "http://${AR_HOST}/v2/${NAME}/manifests/v1"; echo echo "== 4. delete manifest by digest (fully removes it) ==" apicode -X DELETE "http://${AR_HOST}/v2/${NAME}/manifests/${DIGEST}"; echo manifest_gone=$(apicode "http://${AR_HOST}/v2/${NAME}/manifests/${DIGEST}" -H 'Accept: application/vnd.oci.image.manifest.v1+json') echo "manifest-by-digest GET after delete -> $manifest_gone (expect 404)" echo "== 5. delete both blobs ==" apicode -X DELETE "http://${AR_HOST}/v2/${NAME}/blobs/${CONFIG_DIGEST}"; echo apicode -X DELETE "http://${AR_HOST}/v2/${NAME}/blobs/${LAYER_DIGEST}"; echo config_gone=$(apicode -I "http://${AR_HOST}/v2/${NAME}/blobs/${CONFIG_DIGEST}") layer_gone=$(apicode -I "http://${AR_HOST}/v2/${NAME}/blobs/${LAYER_DIGEST}") echo "config blob HEAD after delete -> $config_gone (expect 404)" echo "layer blob HEAD after delete -> $layer_gone (expect 404)" tags_json=$(curl -s -H "Authorization: Bearer $AR_TOKEN" "http://${AR_HOST}/v2/${NAME}/tags/list") echo "tags/list -> $tags_json (expect tags:[])" echo "== 6. THE CHECK: repo-delete on a repo that is empty by every above signal ==" del1=$(api -X DELETE "http://${AR_HOST}/api/v1/${AR_SLUG}/repositories/${REPO}") echo "$del1" status1=$(echo "$del1" | grep -o 'HTTP_STATUS:[0-9]*' | cut -d: -f2) if [[ "$status1" == "204" || "$status1" == "200" ]]; then echo "RESULT: repo delete succeeded immediately - finding did NOT reproduce (may be fixed, or timing-dependent)." exit 0 fi echo "== 7. wait 90s (rule out simple async GC lag) and retry ==" sleep 90 del2=$(api -X DELETE "http://${AR_HOST}/api/v1/${AR_SLUG}/repositories/${REPO}") echo "$del2" status2=$(echo "$del2" | grep -o 'HTTP_STATUS:[0-9]*' | cut -d: -f2) echo echo "=== SUMMARY ===" echo "manifest-by-digest GET (expect 404): $manifest_gone" echo "config blob HEAD (expect 404): $config_gone" echo "layer blob HEAD (expect 404): $layer_gone" echo "tags/list (expect tags:[]): $tags_json" echo "repo-delete attempt 1 status (expect 204/200): $status1" echo "repo-delete attempt 2 status after 90s (expect 204/200): $status2" if [[ "$status2" != "204" && "$status2" != "200" ]]; then echo "REPRODUCED: repo-delete still refuses ($status2) despite a fully-empty repo per the OCI data plane, even after a 90s wait." else echo "NOT REPRODUCED on attempt 2 (resolved after ~90s - would indicate slow-async GC rather than a permanent wedge)." fi ``` </details> ## Environment - AR **version 1.181.0**, commit `f9b0b10d6a55f197a712f6805f55d5494264f2fa` (stable across the run; no auto-update window). - Client: regctl 0.11.5 (cross-checked crane 0.21.7). Finding is client-neutral (data-plane deletes + curl to the management API). - Finder run `SEED=1864315411`, shard `1864315411-7`. ## Verification Independently reproduced by a black-box skeptic in a fresh namespace (deterministic); it specifically ruled out the async-GC explanation by observing the `GET artifacts_count=0` self-contradiction, and confirmed a never-populated repo deletes cleanly (204). White-box root-cause traced the `NO ACTION` FK and the disconnected counter subsystem above.