## Summary The Container Registry service returns HTTP 5xx errors for client-side issues (failed authentication, missing manifests, cancelled uploads), rather than appropriate 4xx client error responses. This causes SLO violation alerts that are not actionable by operators. ## Problem On a GitLab Dedicated tenant, we have observed recurring `RegistryServiceServerErrorSLOViolation` alerts caused by customer automation attempting to mirror third-party container images. When this automation encounters issues (authentication failures, missing content, cancelled requests), the registry returns 500 errors instead of appropriate 4xx responses. This has resulted in: - 3 separate incidents over 4 months (Nov 2025, Mar 2026 x2) - False SLO breach alerts paging on-call engineers - No actionable remediation available to operators - **We have had to silence registry error alerts for this tenant**, reducing our ability to detect actual service issues ## Expected behavior Client-side errors should return appropriate 4xx status codes (401 Unauthorized, 403 Forbidden, 404 Not Found, 400 Bad Request) so they: 1. Do not count against server error SLO budgets 2. Do not trigger operator alerts for issues outside our control 3. Provide clear feedback to clients about what went wrong ## Example log ```json { "time": "2026-03-18T02:45:15.333Z", "kubernetes": { "container_name": "registry", "pod_name": "gitlab-registry-7f98cffc4f-tbsg4", "container_image": "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v4.33.0-gitlab" }, "correlation_id": "4668b8fdb8eabe2416f5d326d0aab9ce", "duration_ms": 2971, "level": "info", "method": "PUT", "msg": "access", "status": 500, "uri": "/v2/<group>/<project>/registry-mirrors/<vendor>/<image>/blobs/uploads/<uuid>?_state=<redacted>&digest=sha256:<hash>", "user_agent": "cosign/v2.4.3 (linux; amd64) go-containerregistry/v0.20.3", "written_bytes": 238, "error": null } ``` Note: `status: 500` but `error: null` - no error detail is provided, making investigation difficult. From other logs in the same timeframe, we see errors like: - `manifest content missing from storage backend` - `s3aws: context canceled` - `s3aws: Path not found` These are all client-induced conditions that should not be 5xx errors. ## Relevant incidents - https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/2266 - https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/2947 - https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/incident-management/-/issues/2996 ## Impact - Repeated false-positive pages for on-call engineers - Incident management overhead for non-actionable alerts - Reduced observability due to silenced alerts - Customer-caused issues appearing as platform reliability problems #### Root Cause Analysis & Implementation Plan ## Root Cause Analysis The issue stems from the error handling middleware in `registry/api/errcode/errors.go`, specifically the `FromUnknownError()` function which serves as a catch-all for unhandled error types. ### Error Handling Flow ``` Request → Handler → Error Occurs ↓ Check Error Type ↓ ┌─────────────┬──────────────┬──────────────┐ ↓ ↓ ↓ ↓ Specific Network Database Fallback Handler Error Error (500) ✗ (4xx/5xx) (503) (503) ``` ### Current Handlers in `FromUnknownError()` The function currently handles: 1. **Network Timeout** → 503 Service Unavailable 2. **Connection Reset** → 400 Bad Request 3. **Context Cancelled** → 400 Bad Request (only unwrapped) 4. **Database Connection Error** → 503 Service Unavailable 5. **GCS Service Unavailable** → 503 Service Unavailable 6. **Everything Else** → 500 Internal Server Error ✗ ### Affected Error Cases #### 1. Manifest Missing from Storage **Error Type:** `distribution.ErrManifestEmpty` **Error Message:** `"manifest content missing from storage backend name=<name> digest=<digest>"` **Current Behavior:** Returns 500 (ErrorCodeUnknown) **Expected Behavior:** Should return 404 Not Found **Code Location:** `errors.go` lines 78-82 **Why It's a Client Error:** The manifest blob is missing from storage, which means either: * The client uploaded an incomplete manifest * The client is trying to access a manifest that was deleted * The storage backend lost the data (rare, but should be 503 not 500) #### 2. Context Cancelled (S3 Wrapped) **Error Type:** `context.Canceled` (wrapped in S3 error) **Error Message:** `"s3aws: context canceled"` **Current Behavior:** Returns 500 (ErrorCodeUnknown) when wrapped in S3 errors **Expected Behavior:** Should return 400 Bad Request **Code Location:** `registry/api/errcode/errors.go` lines 313-315 (only handles unwrapped `context.Canceled`) **Why It's a Client Error:** The client cancelled the request (dropped connection, timeout on client side, etc.) #### 3. S3 Path Not Found **Error Type:** S3 storage driver error **Error Message:** `"s3aws: Path not found"` **Current Behavior:** Returns 500 (ErrorCodeUnknown) **Expected Behavior:** Should return 404 Not Found **Code Location:** No specific handler exists **Why It's a Client Error:** The client is trying to access a blob/manifest that doesn't exist in S3 ## Code Path Analysis ### Blob Upload Handler (`registry/handlers/blobupload.go` lines 393-414) ```go if err == distribution.ErrBlobUploadUnknown { return http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) { buh.Errors = append(buh.Errors, v2.ErrorCodeBlobUploadUnknown.WithDetail(err)) }) } return http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) { buh.Errors = append(buh.Errors, errcode.ErrorCodeUnknown.WithDetail(err)) // <-- 500! }) ``` The handler only checks for `ErrBlobUploadUnknown`. All other errors fall through to the generic `ErrorCodeUnknown` which returns 500. ### Manifest Handler (`registry/handlers/manifests.go`) Uses `errcode.FromUnknownError()` as fallback for unhandled errors, which also defaults to 500. ## Implementation Plan ### Phase 1: Error Handler Enhancements (Minimal Viable Path) Add specific handlers in `FromUnknownError()` for client-side errors: #### 1. Handle ErrManifestEmpty (404 Not Found) ``` var manifestEmpty distribution.ErrManifestEmpty if errors.As(err, &manifestEmpty) { return ErrorCodeManifestUnknown.WithDetail(err) } ``` #### 2. Handle Wrapped Context Cancellation (400 Bad Request) ```go // Unwrap S3 errors to check for context.Canceled var sdErr storagedriver.Error if errors.As(err, &sdErr) { if errors.Is(sdErr.Enclosed, context.Canceled) { return ErrorCodeRequestCanceled.WithDetail(err) } } ``` #### 3. Handle S3 Path Not Found (404 Not Found) ```go var pathNotFound storagedriver.PathNotFoundError if errors.As(err, &pathNotFound) { return ErrorCodeBlobUnknown.WithDetail(err) } ``` ### Phase 2: Blob Upload Handler Improvements Enhance `blobUploadHandler.ResumeBlobUpload()` to catch more specific errors before falling back to generic handler. ### Phase 3: Testing ## Validation Steps ### 1. Unit Test Coverage **File:** `registry/api/errcode/errors_test.go` Add test cases to `TestFromUnknownError`: ```go { name: "manifest empty", err: distribution.ErrManifestEmpty{ Name: "test/repo", Digest: "sha256:abc123", }, expected: http.StatusNotFound, }, { name: "context canceled", err: context.Canceled, expected: http.StatusBadRequest, }, { name: "s3 path not found", err: storagedriver.PathNotFoundError{Path: "/test/path"}, expected: http.StatusNotFound, }, ``` ### 2. Integration Test Coverage **File:** `registry/handlers/blobupload_integration_test.go` Add tests for: * Blob upload with manifest missing * Blob upload with client cancellation * Blob upload with S3 path not found ### 3. Minimal Reproducibility Path (No AWS Dependencies) Create a standalone test that: 1. Directly calls `FromUnknownError()` with mock error types 2. Verifies HTTP status codes returned 3. Runs locally with `go test` **Example:** ``` cd registry/api/errcode go test -v -run TestFromUnknownError ``` ## Testing Strategy ### Unit Tests (No Dependencies) * Mock error types directly using Go's error wrapping * Test `FromUnknownError()` function in isolation * Verify HTTP status codes returned ### Integration Tests (Mocked S3) * Use existing test infrastructure in `testutil` * Mock S3 responses without AWS dependencies * Verify end-to-end behavior ### Local Reproducibility ```sh # Run unit tests go test -v ./registry/api/errcode -run TestFromUnknownError # Run integration tests go test -v -tags=integration,handlers_test ./registry/handlers -run TestBlobUpload ```