"Bad Gateway" using Pages for a merge request artifacts on Omnibus

Summary

When using the expose_as: feature for Pages with a file that is large enough to trigger chunked delivery, nginx returns Bad Gateway for the pages URL due to an invalid Content-Length header.

Steps to reproduce

docker run -itp 80:80 --shm-size=256m --name=gitlab gitlab/gitlab-ce
docker exec -it gitlab bash to get a shell, in which you can apt update && apt install vim to get an editor.
Inside the docker exec shell, vim /etc/gitlab/gitlab.rb and make the following settings changes:
- external_url needs to be set correctly for your environment, because the new "runner registration flow" in 16.x will attempt to redirect you to whatever it thinks its hostname is. I used test-gitlab.
- pages_external_url set to a different domain name than external_url. I used dobby, in loving tribute to what tracking this bug down felt like
- gitlab_pages['enable'] = true
restart the container with docker restart gitlab
Adjust your local DNS or /etc/hosts such that whatever names you made up for external_url and pages_external_url all point at your local machine. You will need to add extra hosts entries for root.<your_pages_external_url> and projects.<your_pages_external_url>. With my particular URL settings, I added this hosts entry:
```
127.0.0.1 test-gitlab dobby root.dobby projects.dobby
```
Register a gitlab runner. I used docker run --rm -it gitlab-runner for this.
Make a test project which just ensures Gitlab Pages are working and enabled.
Try to make a merge request against that test project that has this code in .gitlab-ci.yaml (or just push my test project on .com up

pages for merge requests:
  stage: build
  image: python:3
  script:
  - mkdir public
  - echo "<h1>Hello world</h1><pre>" > public/index.html
  - python -c 'print(repr(list(range(3*1024))).encode("ascii", "xmlcharrefreplace").decode("ascii"))' >> public/index.html
  - echo "</pre>" >> public/index.html
  artifacts:
    expose_as: "demonstration pages site"
    paths:
    - public/index.html
  rules:
  - if: $CI_PIPELINE_SOURCE == "merge_request_event"

The merge request page will offer to show you some artifacts. Click this button to view the artifacts in Pages.

Example Project

Note that this bug does not reproduce on .com, however this project can be pushed up to a test omnibus instance to reproduce the issue.

What is the current bug behavior?

502 Bad Gateway message from nginx

What is the expected correct behavior?

You should see an equivalent to this on your local instance.

Relevant logs and/or screenshots

Pages logs

{
  "level": "info",
  "msg": "http: invalid Content-Length of \"-1\"",
  "time": "2023-06-30T16:10:06Z"
}
{
  "content_type": "text/html; charset=utf-8",
  "correlation_id": "01H46H38CYTRQNVN88XNRMZXB3",
  "duration_ms": 123,
  "host": "test-gitlab",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "pages_https": true,
  "proto": "HTTP/1.0",
  "referrer": "http://test-gitlab/",
  "remote_addr": "REDACTED",
  "remote_ip": "REDACTED",
  "status": 200,
  "system": "http",
  "time": "2023-06-30T16:10:06Z",
  "ttfb_ms": 123,
  "uri": "/-/expose_as-test-project/-/jobs/3/artifacts/public/index.html",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
  "written_bytes": 12158
}

The NGINX logs

2023/06/30 16:27:32 [error] 1725080#0: *695610 upstream sent invalid "Content-Length" header: "Content-Length: -1" while reading response header from upstream, client: REDACTED, server: ~^(?<group>.*)\.dobby$, request: "GET /-/expose_as-test-project/-/jobs/3/artifacts/public/index.html HTTP/2.0", upstream: "http://127.0.0.1:8090/-/expose_as-test-project/-/jobs/3/artifacts/public/index.html", host: "root.dobby"

The api_json.log entry

{
  "time": "2023-06-30T21:35:52.695Z",
  "severity": "INFO",
  "duration_s": 0.14194,
  "db_duration_s": 0.04602,
  "view_duration_s": 0.09592,
  "status": 200,
  "method": "GET",
  "path": "/api/v4/projects/root%2Fexpose_as-test-project/jobs/14/artifacts/public/index.html",
  "params": [],
  "host": "test-gitlab",
  "remote_ip": "REDACTED, REDACTED, 127.0.0.1",
  "ua": "Go-http-client/1.1",
  "route": "/api/:version/projects/:id/jobs/:job_id/artifacts/*artifact_path",
  "user_id": 1,
  "username": "root",
  "token_type": "OauthAccessToken",
  "token_id": 3,
  "queue_duration_s": 0.026851,
  "redis_calls": 8,
  "redis_duration_s": 0.002183,
  "redis_read_bytes": 1418,
  "redis_write_bytes": 670,
  "redis_feature_flag_calls": 7,
  "redis_feature_flag_duration_s": 0.001635,
  "redis_feature_flag_read_bytes": 1418,
  "redis_feature_flag_write_bytes": 564,
  "redis_shared_state_calls": 1,
  "redis_shared_state_duration_s": 0.000548,
  "redis_shared_state_write_bytes": 106,
  "db_count": 8,
  "db_write_count": 0,
  "db_cached_count": 0,
  "db_replica_count": 0,
  "db_primary_count": 8,
  "db_main_count": 5,
  "db_ci_count": 3,
  "db_main_replica_count": 0,
  "db_ci_replica_count": 0,
  "db_replica_cached_count": 0,
  "db_primary_cached_count": 0,
  "db_main_cached_count": 0,
  "db_ci_cached_count": 0,
  "db_main_replica_cached_count": 0,
  "db_ci_replica_cached_count": 0,
  "db_replica_wal_count": 0,
  "db_primary_wal_count": 0,
  "db_main_wal_count": 0,
  "db_ci_wal_count": 0,
  "db_main_replica_wal_count": 0,
  "db_ci_replica_wal_count": 0,
  "db_replica_wal_cached_count": 0,
  "db_primary_wal_cached_count": 0,
  "db_main_wal_cached_count": 0,
  "db_ci_wal_cached_count": 0,
  "db_main_replica_wal_cached_count": 0,
  "db_ci_replica_wal_cached_count": 0,
  "db_replica_duration_s": 0,
  "db_primary_duration_s": 0.011,
  "db_main_duration_s": 0.008,
  "db_ci_duration_s": 0.003,
  "db_main_replica_duration_s": 0,
  "db_ci_replica_duration_s": 0,
  "cpu_s": 0.134418,
  "mem_objects": 43611,
  "mem_bytes": 11513995,
  "mem_mallocs": 33403,
  "mem_total_bytes": 13258435,
  "pid": 15514,
  "worker_id": "puma_16",
  "rate_limiting_gates": [],
  "correlation_id": "01H473QR5S48Y7RT17J71B8GP3",
  "meta.caller_id": "GET /api/:version/projects/:id/jobs/:job_id/artifacts/*artifact_path",
  "meta.remote_ip": "172.17.0.2",
  "meta.feature_category": "build_artifacts",
  "meta.user": "root",
  "meta.user_id": 1,
  "meta.project": "root/expose_as-test-project",
  "meta.root_namespace": "root",
  "meta.client_id": "user/1",
  "request_urgency": "low",
  "target_duration_s": 5,
  "response_bytes": 2
}

gitlab-workhorse

{
  "archive": "/var/opt/gitlab/gitlab-rails/shared/artifacts/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2023_06_30/14/31/artifacts.zip",
  "correlation_id": "01H473QR5S48Y7RT17J71B8GP3",
  "entry": "cHVibGljL2luZGV4Lmh0bWw=\n",
  "level": "info",
  "msg": "SendEntry: sending",
  "path": "/api/v4/projects/root/expose_as-test-project/jobs/14/artifacts/public/index.html",
  "time": "2023-06-30T21:35:52Z"
}
{
  "content_type": "text/html; charset=utf-8",
  "correlation_id": "01H473QR5S48Y7RT17J71B8GP3",
  "duration_ms": 196,
  "host": "test-gitlab",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "",
  "remote_addr": "127.0.0.1:0",
  "remote_ip": "127.0.0.1",
  "route": "^/api/",
  "status": 200,
  "system": "http",
  "time": "2023-06-30T21:35:52Z",
  "ttfb_ms": 195,
  "uri": "/api/v4/projects/root%2Fexpose_as-test-project/jobs/14/artifacts/public/index.html",
  "user_agent": "Go-http-client/1.1",
  "written_bytes": 17356
}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

docker exec -i gitlab gitlab-rake gitlab:env:info | tee >(xclip)

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	3.0.6p216
Gem Version:	3.4.13
Bundler Version:2.4.14
Rake Version:	13.0.6
Redis Version:	6.2.11
Sidekiq Version:6.5.7
Go Version:	unknown

GitLab information
Version:	16.1.1
Revision:	9ce736bb2cd
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	13.11
URL:		http://test-gitlab
HTTP Clone URL:	http://test-gitlab/some-group/some-project.git
SSH Clone URL:	git@test-gitlab:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.23.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

docker exec -i gitlab gitlab-rake gitlab:check SANITIZE=true | tee >(xclip)

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
1/1 ... yes
1/2 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

~~I'll edit this post if I find any. My team spent the better part of today just figuring out this much.~~

The problem began when we upgraded our real instance to 16.1.1 yesterday (from 16.0.4).

Edit: Thanks to @stargo (see comments) we now have a workaround.

# /etc/gitlab/gitlab.rb
pages_nginx['custom_gitlab_server_config'] = 'proxy_http_version 1.1;'

Edited Jul 05, 2023 by Janis Altherr