## :fire: Problem The GitLab Conan package registry cannot communicate meaningful error messages to Conan clients (`conan` v1 and v2). Currently, all error responses use `Content-Type: application/json` with a body like `{"message": "403 Forbidden - Package protected."}`. The Conan client's `response_to_str()` function attempts to parse this as [Artifactory-compatible JSON](https://jfrog.com/help/r/jfrog-rest-apis/errors), but since GitLab's format doesn't match, the **parsing silently fails** and the raw JSON string is displayed to users: ``` ERROR: 403 Forbidden. Remote: {"message": "403 Forbidden - Package protected."} ``` When the Artifactory JSON format is used, Conan extracts and formats the message cleanly: ``` ERROR: 403 Forbidden. Remote: 403: Package protected. ``` This affects all upload, authentication, and some download error scenarios. Users see unhelpful raw JSON instead of clean, parsed error messages. ## :bulb: Solution Return [Artifactory-compatible JSON error responses](https://jfrog.com/help/r/jfrog-rest-apis/errors) from all Conan package registry endpoints. When enabled, all 4xx/5xx responses from Conan endpoints will return: - `Content-Type: application/json` (already the default) - A JSON body with the `errors` array format ### :mag: Example ```http HTTP/1.1 403 Forbidden Content-Type: application/json { "errors": [ { "status": 403, "message": "Package protected." } ] } ``` Conan v1/v2 users will then see: ``` ERROR: 403 Forbidden. Remote: 403: Package protected. ``` ### :gear: Design details - Use the Artifactory JSON format: `{"errors":[{"status":N,"message":"..."}]}` — this is the de facto standard for Conan server error responses. - Feature-flagged as `conan_artifactory_error_responses` (default disabled for gradual rollout). - The `message` field MUST NOT leak internal infrastructure details (hostnames, stack traces, etc.). - Extract a clean `message` from the legacy format by stripping the `"NNN StatusPhrase - "` prefix (e.g., `"403 Forbidden - Package protected."` becomes `"Package protected."`). When no detail is present (e.g., bare `forbidden!`), use the HTTP status phrase as the message. ### :shield: Backward compatibility This change is fully backward-compatible: - **Conan v1 and v2 clients already call `response_to_str()`** on error responses. This function checks for `Content-Type: application/json` and attempts to parse `{"errors":[...]}`. Today, parsing fails silently (caught by a bare `except`) and falls back to showing the raw body. With the new format, parsing succeeds and users see clean messages instead. - **Content-Type remains `application/json`** — no change to the response content type. - **Status codes remain unchanged** — clients that only check status codes are unaffected. - **Conan v1.37+ and all v2** handle `Content-Type: application/json; charset=utf-8` correctly after [conan-io/conan#8912](https://github.com/conan-io/conan/pull/8912). Older v1 versions (<1.37) fail on charset parameters, but GitLab's Grape framework returns `application/json` without charset by default, so this is not an issue. ## :link: Client compatibility matrix ### :satellite: How Conan clients handle HTTP errors There are two mechanisms a server can use to communicate error information to Conan clients: 1. **HTTP status code** — always available, all clients use it 2. **Response body** — parsed via `response_to_str()` which extracts Artifactory JSON or falls back to raw text ### :conan: Conan error response parsing (`response_to_str`) The [`response_to_str()`](https://github.com/conan-io/conan/blob/develop2/conans/client/rest/__init__.py) function is the single point where Conan clients extract error details from server responses: ```python def response_to_str(response): content_type = response.headers.get("content-type") if content_type == "application/json": # Artifactory format: {"errors":[{"status":N,"message":"..."}]} data = json.loads(content)["errors"][0] content = "{}: {}".format(data["status"], data["message"]) elif "text/html" in content_type: content = "{}: {}".format(response.status_code, response.reason) return content # raw text for anything else ``` **Key behavior:** - Checks `Content-Type` for exact `"application/json"` match - Parses the **first** error in the `errors` array - Formats as `"status: message"` (e.g., `"403: Package protected."`) - Falls back to raw body content if JSON parsing fails (bare `except` catches all) - HTML responses show only `"status_code: reason_phrase"` ### :package: Error status code to exception mapping Conan maps HTTP status codes to specific exception types via [`get_exception_from_error()`](https://github.com/conan-io/conan/blob/develop2/conans/errors.py): | HTTP Status | Conan Exception | User-visible prefix | |---|---|---| | 400 | `RequestErrorException` | `ERROR: 400 Bad Request` | | 401 | `AuthenticationException` | `ERROR: 401 Unauthorized` | | 403 | `ForbiddenException` | `ERROR: 403 Forbidden` | | 404 | `NotFoundException` | `ERROR: 404 Not Found` | | 500 | `InternalErrorException` | `ERROR: 500 Internal Server Error` | ### :arrow_down: Download operations (FileDownloader) The [FileDownloader](https://github.com/conan-io/conan/blob/develop2/conans/client/downloaders/file_downloader.py) has **hardcoded error handling** for some status codes: | Status | Conan v1 | Conan v2 (2.3.0+) | Body passthrough? | |---|---|---|---| | 404 | `NotFoundException("Not found: %s" % url)` | Same | **No** — hardcoded URL-only message | | 5xx | `ConanException("Error %d downloading file %s")` | Same | **No** — hardcoded status + URL | | 403 | `ForbiddenException(response_to_str(response))` | Same | **Yes** — body parsed | | 401 | `AuthenticationException()` (no body) | `AuthenticationException(response_to_str(response))` | **v2 only** ([conan-io/conan#15983](https://github.com/conan-io/conan/pull/15983)) | ### :arrow_up: Upload and auth operations (REST client) For upload, search, and authentication operations, **all** non-200 responses pass through `response_to_str()`: ```python if response.status_code != 200: raise get_exception_from_error(response.status_code)(response_to_str(response)) ``` This means all 4xx/5xx error bodies are parsed and displayed to users for upload and auth flows. ### :bar_chart: Summary | Operation | Status codes with body passthrough | Notes | |---|---|---| | **Upload** (recipe/package files) | All 4xx/5xx | `response_to_str()` always called | | **Authentication** (`/users/authenticate`) | All 4xx/5xx | `response_to_str()` always called | | **Search** | All 4xx/5xx | `response_to_str()` always called | | **Download** 403 | 403 | Body parsed via `response_to_str()` | | **Download** 401 | 401 (v2 only) | v1 shows generic message; v2.3.0+ parses body | | **Download** 404 | — | Hardcoded: shows URL only | | **Download** 5xx | — | Hardcoded: shows status + URL only | ### :twisted_rightwards_arrows: Conan v1 vs v2 | Feature | Conan v1 | Conan v2 | |---|---|---| | `response_to_str()` | Present (same logic) | Present (same logic) | | Artifactory JSON parsing | Yes | Yes | | Content-Type charset handling | Fixed in v1.37+ ([conan-io/conan#8912](https://github.com/conan-io/conan/pull/8912)) | Yes | | 401 body in downloads | No (generic message) | Yes (v2.3.0+, [conan-io/conan#15983](https://github.com/conan-io/conan/pull/15983)) | | `get_json()` Content-Type check | `== "application/json"` or `"application/json; charset=utf-8"` | Same | References: - [response_to_str (v2)](https://github.com/conan-io/conan/blob/develop2/conans/client/rest/__init__.py) - [FileDownloader (v2)](https://github.com/conan-io/conan/blob/develop2/conans/client/downloaders/file_downloader.py) - [rest_client_common.py (v2)](https://github.com/conan-io/conan/blob/develop2/conans/client/rest/rest_client_common.py) - [errors.py (v2)](https://github.com/conan-io/conan/blob/develop2/conans/errors.py) - [conan-io/conan#8884](https://github.com/conan-io/conan/issues/8884) — Content-Type charset handling issue - [conan-io/conan#15983](https://github.com/conan-io/conan/pull/15983) — 401 error body passthrough in FileDownloader - [conan-io/conan#6193](https://github.com/conan-io/conan/issues/6193) — 403 Artifactory permission error handling ## :wrench: Implementation approach Create a helper module `API::Helpers::Packages::Conan::ArtifactoryErrors` that overrides `render_structured_api_error!` — the single method all error helpers flow through: ``` forbidden!("reason") / not_found!("Package") / bad_request!("reason") / ... → render_api_error_with_reason! or render_api_error! → render_structured_api_error!({ 'message' => "4xx ... - reason" }, status) → error!(hash, status, header) # Grape's method ``` The override: 1. Checks the `conan_artifactory_error_responses` feature flag 2. Only transforms 4xx/5xx responses (not 2xx like `no_content!` which also flows through this path) 3. Extracts a clean `message` from the legacy message string by stripping the `"NNN StatusPhrase - "` prefix 4. Constructs the Artifactory JSON body: `{"errors":[{"status":N,"message":"..."}]}` 5. Calls Grape's `error!` with the transformed body 6. When the flag is disabled, calls `super` — zero behavioral change Include this module in the Conan API classes — no impact on other API endpoints. **Note:** The `users/authenticate` endpoint uses `format :txt` with `content_type :txt, 'text/plain'`. The `unauthorized!` call on this endpoint will still flow through `render_structured_api_error!`, but the Grape `format` directive controls the final Content-Type. This needs special attention during implementation to ensure the Artifactory JSON format is returned with `application/json` Content-Type even on the `users` namespace, or alternatively this endpoint can be excluded since the 401 on this endpoint is a simple "no token" case. ## :file_folder: Impacted files ### :new: New files | File | Purpose | |---|---| | `config/feature_flags/development/conan_artifactory_error_responses.yml` | Feature flag definition | | `lib/api/helpers/packages/conan/artifactory_errors.rb` | Helper module overriding `render_structured_api_error!` | | `spec/lib/api/helpers/packages/conan/artifactory_errors_spec.rb` | Unit tests for the helper | ### :pencil2: Modified files | File | Change | |---|---| | `lib/api/conan/v1/project_packages.rb` | Include the helper module | | `lib/api/conan/v1/instance_packages.rb` | Include the helper module | | `lib/api/conan/v2/project_packages.rb` | Include the helper module | | `spec/requests/api/conan/v1/project_packages_spec.rb` | Add Artifactory error format shared examples + feature flag contexts | | `spec/requests/api/conan/v1/instance_packages_spec.rb` | Same | | `spec/requests/api/conan/v2/project_packages_spec.rb` | Same | | `ee/spec/requests/api/conan/v1/project_packages_spec.rb` | Same | | `spec/support/shared_examples/requests/api/conan_packages_shared_examples.rb` | Add shared examples for Artifactory error response validation | ## :rotating_light: Error scenarios in Conan API ### :arrows_counterclockwise: Shared endpoints (`lib/api/concerns/packages/conan/shared_endpoints.rb`) | Status | Trigger | Expected `message` | |---|---|---| | 400 | `ActiveRecord::RecordInvalid` (package validation) | Validation message (e.g., `"Validation failed: ..."`) | | 400 | `bad_request!(response.message)` (search error) | `"Search term length must be less than 200 characters"` or `"Too many wildcards in search term. Maximum is 5"` | | 401 | `unauthorized!` (no valid token on `/users/authenticate`) | `"Unauthorized"` | | 404 | `not_found!` (FIPS mode enabled) | `"Not Found"` | | 404 | `not_found!('Package')` (search by reference) | `"Package Not Found"` | ### :one: V1 endpoints (`lib/api/concerns/packages/conan/v1_endpoints.rb`) | Status | Trigger | Expected `message` | |---|---|---| | 400 | `bad_request!('File is too large')` | `"File is too large"` | | 400 | `bad_request!(nil)` (invalid JSON body) | `"Bad request"` | | 400 | `bad_request!(service_response.message)` (package/file creation error) | Service error message | | 403 | `forbidden!(service_response.message)` (protected package on create) | Protection error message | | 403 | `forbidden!('Package protected.')` (via `protect_package!`) | `"Package protected."` | | 403 | `forbidden!` (remote storage error) | `"Forbidden"` | | 404 | `not_found!('Package')` (download, package not found) | `"Package Not Found"` | | 404 | `not_found!('Non checksum storage')` (checksum deploy header) | `"Non checksum storage Not Found"` | | 404 | `render_api_error!("No recipe manifest found", 404)` | `"No recipe manifest found"` | ### :two: V2 endpoints (`lib/api/conan/v2/project_packages.rb`) | Status | Trigger | Expected `message` | |---|---|---| | 400 | Same as V1 upload errors | Same as V1 | | 403 | Same as V1 upload errors | Same as V1 | | 404 | `not_found!('Package')` (various endpoints) | `"Package Not Found"` | | 404 | `not_found!('Revision')` (recipe revision not found) | `"Revision Not Found"` | | 404 | `not_found!('Recipe files')` (no files for revision) | `"Recipe files Not Found"` | | 404 | `not_found!('Package files')` (no package files) | `"Package files Not Found"` | | 404 | `not_found!('Package Revision')` (package revision not found) | `"Package Revision Not Found"` | | 422 | `unprocessable_entity!("Cannot delete more than 1000 files")` | `"Cannot delete more than 1000 files"` | ### :lock: Authorization errors (implicit via `route_setting`) | Status | Trigger | Expected `message` | |---|---|---| | 401 | Invalid/missing/expired credentials | `"Unauthorized"` | | 403 | Insufficient permissions (read/upload/delete) | `"Forbidden"` | ## :white_check_mark: Verification plan ### :globe_with_meridians: curl verification (GDK) ```bash # Enable feature flag # Rails console: Feature.enable(:conan_artifactory_error_responses) # 401 - no token curl -v "http://gdk.test:3000/api/v4/projects/1/packages/conan/v1/users/authenticate" # Expected: {"errors":[{"status":401,"message":"Unauthorized"}]} # 403 - protected package upload (V1) curl -v -X PUT -H "Private-Token: $TOKEN" \ "http://gdk.test:3000/api/v4/projects/1/packages/conan/v1/files/pkg/1.0/user/channel/0/export/conanfile.py/authorize" # Expected: {"errors":[{"status":403,"message":"Package protected."}]} # 404 - nonexistent package (V1) curl -v -H "Private-Token: $TOKEN" \ "http://gdk.test:3000/api/v4/projects/1/packages/conan/v1/conans/nonexistent/1.0/user/channel/search" # Expected: {"errors":[{"status":404,"message":"Package Not Found"}]} # 404 - nonexistent package (V2) curl -v -H "Private-Token: $TOKEN" \ "http://gdk.test:3000/api/v4/projects/1/packages/conan/v2/conans/nonexistent/1.0/user/channel/latest" # Expected: {"errors":[{"status":404,"message":"Package Not Found"}]} # 400 - search term too long curl -v -H "Private-Token: $TOKEN" \ "http://gdk.test:3000/api/v4/projects/1/packages/conan/v1/conans/search?q=$(python3 -c 'print("a"*201)')" # Expected: {"errors":[{"status":400,"message":"Search term length must be less than 200 characters"}]} # CRITICAL: verify Content-Type is application/json (no charset parameter for v1 < 1.37 compat) curl -sI -H "Private-Token: invalid" \ "http://gdk.test:3000/api/v4/projects/1/packages/conan/v1/conans/search?q=test" \ | grep -i content-type # Should be: Content-Type: application/json # Verify feature flag OFF returns old format # Rails console: Feature.disable(:conan_artifactory_error_responses) curl -s "http://gdk.test:3000/api/v4/projects/1/packages/conan/v1/users/authenticate" # Expected: {"message":"401 Unauthorized"} ``` ### :package: Client versions to test | Client | Version | Purpose | |---|---|---| | Conan | **v2 (latest)** | Primary test target — full `response_to_str()` support | | Conan | **v2.3.0** | First version with 401 body passthrough in FileDownloader | | Conan | v1 (latest 1.x) | Verify backward compatibility with v1 `response_to_str()` | ### :test_tube: Test scenarios For each client version, test with feature flag **enabled** and **disabled**: 1. **Authentication failure** (401) — wrong/missing token on `conan remote login`; verify error detail displayed 2. **Permission denied** (403) — token without write permission; `conan upload` to project 3. **Protected package** (403) — `conan upload` to protected package; verify `"Package protected."` message 4. **Package not found** (404) — `conan install` nonexistent package reference 5. **Upload too large** (400) — upload file exceeding size limit 6. **Search error** (400) — search with term > 200 characters ### :crystal_ball: Expected output **Conan v2 with flag enabled:** ``` ERROR: 403 Forbidden. Remote: 403: Package protected. ``` **Conan v2 with flag disabled (current behavior):** ``` ERROR: 403 Forbidden. Remote: {"message": "403 Forbidden - Package protected."} ``` **Conan v1 with flag enabled:** ``` ERROR: 403 Forbidden. Remote: 403: Package protected. ``` The behavior improvement is identical for v1 and v2 since both use the same `response_to_str()` logic. The only difference is that v1 FileDownloader doesn't pass through 401 bodies on download — but downloads rarely produce 401s in practice (authentication happens earlier in the flow). ## :link: References - Parent issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/595485 - [Artifactory REST API errors format](https://jfrog.com/help/r/jfrog-rest-apis/errors) - [Conan `response_to_str` source](https://github.com/conan-io/conan/blob/develop2/conans/client/rest/__init__.py) - [conan-io/conan#15983](https://github.com/conan-io/conan/pull/15983) — 401 error body passthrough - [conan-io/conan#8884](https://github.com/conan-io/conan/issues/8884) — Content-Type charset handling - [conan-io/conan#6193](https://github.com/conan-io/conan/issues/6193) — 403 error handling