## Summary This issue is to roll out the fix for [#371991](https://gitlab.com/gitlab-org/gitlab/-/issues/371991) on production, currently behind the `workhorse_download_etag_caching` feature flag introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/238127. When enabled, `API::Helpers#present_carrierwave_file!`, `send_artifacts_entry`, and `legacy_send_artifacts_entry` either emit a content-derived `ETag` (when callers provide one via the `etag:` kwarg) or set `Last-Modified: '0'` to suppress `Rack::ETag`'s bogus empty-body digest. When disabled, the old behaviour is preserved (all Workhorse-delegated downloads share the constant `W/"12ae32cb1ec02d01eda3581b127c1fee"` ETag). Initial opt-in: the four CI artifact endpoints (`/projects/:id/jobs/:job_id/artifacts`, `/projects/:id/jobs/:job_id/artifacts/*artifact_path`, `/projects/:id/jobs/artifacts/:ref_name/download`, `/projects/:id/jobs/artifacts/:ref_name/raw/*artifact_path`) emit `"<file_sha256>"` / `"<sha256(file_sha256:path)>"`. ## Owners - Most appropriate Slack channel to reach out to: `#g_pipeline_execution` - Best individual to reach out to: @marc_shaw ## Expectations ### What are we expecting to happen? - The four CI artifact endpoints start returning correct content-based ETags. Repeated `If-None-Match` requests with the same ETag get `304`; with a different one, `200`. - Every other Workhorse-delegated API download (avatars, exports, all package formats, LFS, vulnerability/dependency exports, virtual registries, …) stops emitting the bogus `W/"12ae32cb1ec02d01eda3581b127c1fee"` ETag. They emit no ETag and `Last-Modified: '0'` instead, which prevents spurious `304`s. - Clients that were (erroneously) caching against the bogus constant invalidate their cache and start receiving `200`s with the actual response body — which is what they should have been receiving all along. ### What can go wrong and how would we detect it? - A client that was relying on `304 Not Modified` to skip re-downloading an artifact will see `200 OK` on the first request after rollout — that's a one-time cache invalidation, not a regression. - A misbehaving intermediate cache that strips response bodies on `304` could conceivably cause confusion for the brief window during rollout. Monitor `api_ci_artifacts_download` and `api_packages_*` request rates and 4xx/5xx error rates in https://dashboards.gitlab.net. ## Rollout Steps ### Rollout on non-production environments - [ ] Verify the introducing MR is merged to `master` and deployed. - [ ] `/chatops gitlab run feature set workhorse_download_etag_caching 50 --actors --dev --pre --staging --staging-ref` - [ ] Monitor 4xx/5xx rates on artifact + package download endpoints. - [ ] `/chatops gitlab run feature set workhorse_download_etag_caching true --dev --pre --staging --staging-ref` ### Specific rollout on production - [ ] `/chatops gitlab run feature set workhorse_download_etag_caching --user=marc_shaw true` - [ ] `/chatops gitlab run feature set workhorse_download_etag_caching --feature-group=gitlab_team_members true` - [ ] Verify on a personal project: pull an artifact, replay with `If-None-Match` of the returned ETag, expect `304`. - [ ] `/chatops gitlab run feature set workhorse_download_etag_caching 25` — monitor. - [ ] `/chatops gitlab run feature set workhorse_download_etag_caching 50` — monitor. - [ ] `/chatops gitlab run feature set workhorse_download_etag_caching 100` — monitor. ### Final steps - [ ] Remove `workhorse_download_etag_caching` references from the codebase and delete its YAML once rollout completes.