Pages artifact browsing broken (403/empty 200) by artifacts_server_url deriving from internal_gitlab_server

Summary

Browsing job artifact files through the GitLab Pages artifacts server fails on every request on Omnibus 19.2.0+ when gitlab_pages['internal_gitlab_server'] points directly at Puma (for example http://localhost:8080) and gitlab_pages['artifacts_server_url'] is not explicitly set. Users get HTTP 403 {"message":"403 Forbidden"} on 19.2.1+ or HTTP 200 with an empty body on 19.2.0. Regular Pages sites keep working, which makes the failure hard to attribute.

The cause is a changed cookbook default: since 19.2.0, artifacts_server_url derives from internal_gitlab_server instead of gitlab_server, so the Pages daemon sends artifact API requests directly to Puma, bypassing NGINX and Workhorse. Artifact file responses use the Workhorse senddata mechanism, so a Puma-direct target can never serve them.

Commit b035bf0b9fca changed the derivation in files/gitlab-cookbooks/gitlab-pages/libraries/gitlab_pages.rb:19.1.x (gitlab_pages.rb#L85):

Gitlab['gitlab_pages']['artifacts_server_url'] ||= Gitlab['gitlab_pages']['gitlab_server'].chomp('/') + '/api/v4'

19.2.0+ (gitlab_pages.rb#L93):

Gitlab['gitlab_pages']['artifacts_server_url'] ||= (Gitlab['gitlab_pages']['internal_gitlab_server'] || Gitlab['gitlab_pages']['gitlab_server']).chomp('/') + '/api/v4'

GitLab's Pages troubleshooting documentation recommends this value: Unable to access GitLab Pages tells administrators behind a TLS-terminating load balancer to set gitlab_pages['internal_gitlab_server'] = 'http://localhost:8080'. For installations following that advice, the rendered daemon config flips from artifacts-server=https://<external_url>/api/v4 to artifacts-server=http://localhost:8080/api/v4.

Three customers hit this within a month of the 19.2.0 release (internal tickets in References).

  1. 403 variant, upgrade 19.1.2 => 19.2.1
  2. Empty-200 variant, 19.2.0 
  3. 403 after successful OAuth

Steps to reproduce

  1. Omnibus 19.2.1+, pages_external_url set, gitlab_pages['access_control'] = true, gitlab_pages['artifacts_server'] = true (default), gitlab_pages['internal_gitlab_server'] = 'http://localhost:8080', gitlab_pages['artifacts_server_url'] unset.
  2. gitlab-ctl reconfigure; confirm grep '^artifacts-server=' /var/opt/gitlab/gitlab-pages/gitlab-pages-config shows http://localhost:8080/api/v4.
  3. In a project with job artifacts, use Build > Jobs > Job artifacts > Browse and open any HTML/text/JSON file (the "You are being redirected away from GitLab" flow to the Pages domain).

Current behavior

HTTP 403 {"message":"403 Forbidden"} (19.2.1+/19.1.3+/19.0.5+) or HTTP 200 with empty body (19.2.0) for every artifact file served via the Pages domain, including after successful OAuth authentication. Regular Pages sites keep working, which makes the failure hard to attribute. The condition is deterministic: it reproduces on every request whenever internal_gitlab_server targets Puma directly, artifacts_server_url is unset, and the package is 19.2.0+. It also surfaces without any upgrade, when an affected instance enables Pages access control for the first time (that is what routes private-project artifact previews through the Pages daemon).

Expected behavior

Artifact browsing through Pages keeps working across the upgrade, or at minimum the behavior change is documented and called out in the changelog and the Pages administration docs.

Proposed fixes (any of)

  1. Revert the artifacts_server_url part of b035bf0b. Preferring internal_gitlab_server is only safe when it fronts Workhorse, which the cookbook cannot detect, so deriving from gitlab_server as before is the safer default.
  2. Documentation and changelog note: if internal_gitlab_server points directly at Puma, artifacts_server_url must be pinned explicitly to a Workhorse-fronted URL.
  3. Longer term (gitlab-pages/Rails): fail loudly when a senddata response cannot be materialized or verified (5xx with a log hint) instead of an empty 200 or an opaque 403 after successful authentication.

Workaround

Pin the value explicitly on every node running the Pages daemon (explicit settings win over the ||= default); internal_gitlab_server can be kept for the daemon's auth and API calls:

gitlab_pages['artifacts_server_url'] = 'https://<gitlab-external-url>/api/v4'
sudo gitlab-ctl reconfigure

Any URL that reaches NGINX/Workhorse works, and per-node values are valid (needed for example behind AWS NLBs whose targets cannot hairpin to themselves). Alternatively remove gitlab_pages['internal_gitlab_server'].