Intermittent file upload error to the issue comment box
Customer is running GitLab on Kubernetes with Object Storage. When dragging and dropping an image file to the issue comment box error appears in the UI Internal Server Error
. And the below error appears in the workhorse log, this happens intermitently and succeeds on the second upload attempt. Below out of 2 image uploads one failed.
host.com<http://host.com> sanitized IP - - [2019/10/28:17:13:05 +0000] "GET /user/test-project2/issues/2/realtime_changes HTTP/1.1" 304 0 "https://host.com/user/test-project2/issues/2" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36" %!f(int64=015)
time="2019-10-28T17:13:07Z" level=info msg="running exiftool to remove any metadata" correlation_id=PLMhpfoYex2 filename=c300_5.JPG
time="2019-10-28T17:13:07Z" level=info msg="running exiftool to remove any metadata" correlation_id=r656zMAN564 filename=c300_2.JPG
time="2019-10-28T17:13:07Z" level=error msg=error correlation_id=PLMhpfoYex2 error="handleFileUploads: extract files from multipart: persisting multipart file: read |0: file already closed" method=POST uri=/user/test-project2/uploads
host.com<http://host.com> sanitized IP - - [2019/10/28:17:13:07 +0000] "POST /user/test-project2/uploads HTTP/1.1" 500 22 "https://host.com/user/test-project2/issues/2" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36" %!f(int64=873)
Object Storage settings:
object storage on GCS buckets. In helm chart values file the settings are:
--set global.appConfig.artifacts.bucket=bucket-name
--set global.appConfig.artifacts.connection.secret=gitlab-storage
--set global.appConfig.artifacts.connection.key=connection
That secret and key contains:
provider: Google
google_project: <project>
google_client_email: dev-gitlab-object-storage@<project>.iam.gserviceaccount.com<http://iam.gserviceaccount.com>
# Ensure indentation is correct for `google_json_key_string`
# - YAML is a superset of JSON, so you can paste & indent
google_json_key_string: |
<JSON content of google service account key>
A little more context on the issue:
The customer is in the process of migrating an Omnibus installation on a VM to Kubernetes cluster running GKE on-prem. The omnibus installation they migrated from had object storage enabled prior to migrating and the issue is not reproducible there. It only happens on some uploads and typically if they retry it will go through on a second attempt. If uploading a batch of files, typically, some of them will upload and it will error out somewhere in the middle of the batch. Most of the images are around 4-5MB in size.