Workhorse prematurely closing connection when file not found in artifact

nginx error reports

I noticed it while investigating a different problem:

The nginx error logs on the api hosts don't have much in them, except for a similar message that is repeated a fair bit:

2020/09/01 05:20:58 [error] 21074#0: *134121040 upstream prematurely closed connection while reading upstream, client: ..., server: gitlab.com, request: "GET /api/v4/projects/.../jobs/.../artifacts/assets/style.css HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/api/v4/projects/.../jobs/.../artifacts/assets/style.css", host: "int.gprd.gitlab.net"

2020/09/01 04:55:40 [error] 21074#0: *133670758 upstream prematurely closed connection while reading upstream, client: ..., server: gitlab.com, request: "GET /api/v4/projects/.../jobs/.../artifacts/.../datatables.bootstrap4.min.js HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/api/v4/projects/.../jobs/.../artifacts/.../datatables.bootstrap4.min.js", host: "int.gprd.gitlab.net"

2020/09/01 00:22:49 [error] 21077#0: *128822499 upstream prematurely closed connection while reading upstream, client: ..., server: gitlab.com, request: "GET /api/v4/projects/.../jobs/.../artifacts/... HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/api/v4/projects/.../jobs/.../artifacts/...", host: "gitlab.com"

Looking for correlations with the Workhorse logs, I notice that each matches an error message log from workhorse, along these lines:

gitlab-zip-cat error: zip entry not found - find "ci-version.txt" in "https://storage.googleapis.com/gitlab-gprd-artifacts/.../artifacts.zip?GoogleAccessId=...&Signature=[FILTERED]&Expires=1598941144": not found, code: 11

Hypothesis

When workhorse is unable to find an entry in an artifact zip file, it closes the connection prematurely, causing nginx to log an error.