Uploading empty file returns 500 error
Summary
When using the API to upload a file to a project, if the file is empty, GitLab returns a 500 error. This 500 error is caused by the application getting a 400 error from the object storage.
This was first pointed out to me by a customer, who noted that GitLab seemed to be mangling the Content-Type when passing the file along to the object storage.
So far, I have only seen this when using the Helm chart with S3-compatible storage. It may be that this is a bug in all of workhorse (or some other component, not the Helm chart), but I have yet to see it reproduced elsewhere. We can move the issue if we find that it is not specific to the chart.
Related support ticket
This was pointed out to me by a large customer with a premium subscription, in a support ticket (internal).
Steps to reproduce
- Set up a minimal Helm chart deployment of GitLab, using the default minio object storage.
- Create a project to upload to.
- Run the following:
$ touch emtpy-test.txt $ curl --request POST --header "PRIVATE-TOKEN: $PAT" --form "file=@empty-test.txt" "https://gitlab.example.com/api/v4/projects/2/uploads" Internal server error
I was not able to reproduce this on:
- GitLab.com
- Omnibus
- with Google Cloud Storage
- with Amazon S3
Configuration used
My own testing was fairly minimal:
certmanager-issuer:
email: support@gitlab.com
gitlab-runner:
runners:
privileged: true
global:
hosts:
domain: example.gitlabsandbox.net
externalIP: xx.xx.xx.xx
kas:
enabled: false
Current behavior
Uploading an empty file returns a 500 error.
Expected behavior
I expect the empty file to properly upload (as it does on GitLab.com, and an Omnibus install with Amazon S3 or Google Cloud Storage).
Versions
- Chart: 6.1.2 (14.6.7)
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client: 1.24.2
- Server: 1.22.8-gke.202
- Helm:
- Client: v3.9.0
- Object storage: included minio
The customer I was working with also saw this with the following:
- Chart: 5.6.7 (15.1.2)
- Helm:
- Client: v3.7.1
- Object storage: Dell EMC 3.6.2.2
Relevant logs
The failures from the workhorse container:
{"correlation_id":"01G82H2AW8R4M7ZGJZBZ89QW4W","error":"handleFileUploads: extract files from multipart: persisting multipart file: CompleteMultipartUpload request http://gitlab-minio-svc.default.svc:9000/gitlab-uploads/tmp/uploads/1657942387-34-0001-8569-9b498b33a4994432eb2dcdf74f5c5944?uploadId=0a14eb6f-08d7-4479-b0e7-58e8555603fc\u0026X-Amz-Expires=15300\u0026X-Amz-Date=20220716T033308Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=BSxNPDE1SoIhUgc4MIylA1e1XiLSO7GRFWd56qEup9PIa9eIk4kr6dbc1cKQWUBV%2F20220716%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=content-type%3Bhost\u0026X-Amz-Signature=[FILTERED] returned: 400 Bad Request","level":"error","method":"POST","msg":"","time":"2022-07-16T03:33:08Z","uri":"/api/v4/projects/2/uploads"}
{"content_type":"text/plain; charset=utf-8","correlation_id":"01G82H2AW8R4M7ZGJZBZ89QW4W","duration_ms":698,"host":"example.gitlabsandbox.net","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"xx.xx.xx.xx:50450","remote_ip":"xx.xx.xx.xx","route":"^/api/v4/projects/[^/]+/uploads\\z","status":500,"system":"http","time":"2022-07-16T03:33:08Z","ttfb_ms":696,"uri":"/api/v4/projects/2/uploads","user_agent":"curl/7.84.0","written_bytes":22}
Trimming down the messages and doing some formatting, we get something like:
Error 500: handleFileUploads: extract files from multipart: persisting multipart file: CompleteMultipartUpload request http://minio.bucket.svc/upload/path/example returned: 400 Bad Request
Workaround
It may be acceptable to ask customers not to upload empty files while this issue exists.