Uploads are not working on pre.gitlab.com
On upload we see the following:
{"correlation_id":"IeWPvzQdIh4","error":"handleFileUploads: extract files from multipart: persisting multipart file: uploadLocalFile: mkdir \"/mnt/storage/uploads/uploads/tmp/uploads\": mkdir /mnt/storage/uploads: permission denied","level":"error","method":"POST","msg":"error","time":"2019-07-02T06:38:40Z","uri":"/jarv/registry-test/uploads"}
Designs
- Show closed items
Relates to
- gitlab-org/gitlab-foss #47220
- gitlab-org/gitlab #2254712.5
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- John Jarvis assigned to @jarv
assigned to @jarv
- Author Owner
uploads config:
gitlab_rails['uploads_object_store_enabled'] = true gitlab_rails['uploads_object_store_remote_directory'] = "gitlab-pre-uploads" gitlab_rails['uploads_object_store_background_upload'] = true gitlab_rails['uploads_object_store_proxy_download'] = true gitlab_rails['uploads_object_store_connection'] = {"provider"=>"Google", "google_project"=>"gitlab-pre", "google_client_email"=>"gitlab-object-storage@gitlab-pre.iam.gserviceaccount.com", "google_json_key_location"=>"/etc/gitlab/gcs-creds.json"}
- Author Owner
the uploads directory is set to:
gitlab_rails['uploads_storage_path'] = "/mnt/storage/uploads"
in gitlab.rb, we have object storage enabled but direct upload is not:
gitlab_rails['artifacts_object_store_direct_upload'] = true gitlab_rails['lfs_object_store_direct_upload'] = true gitlab_rails['packages_object_store_direct_upload'] = true
Edited by John Jarvis - Author Owner
- Author Owner
Even after enabling direct uploads we were seeing this error:
{"correlation_id":"Kk0EeZD4XF8","duration":0.004821979,"host":"pre.gitlab.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referer":"","remoteAddr":"127.0.0.1:0","remoteIp":"127.0.0.1","status":200,"system":"http","time":"2019-07-02T07:06:17Z","uri":"/-/health","userAgent":"","written":9} {"correlation_id":"WQOEf1K1qR3","error":"handleFileUploads: extract files from multipart: persisting multipart file: uploadLocalFile: mkdir \"/mnt/storage/uploads/uploads/tmp/uploads\": mkdir /mnt/storage/uploads: permission denied","level":"error","method":"POST","msg":"error","time":"2019-07-02T07:06:17Z","uri":"/jarv/registry-test/uploads"}
The only way I could fix this was to manually create the dir
/mnt/storage/uploads
- Author Owner
I believe the omnibus should be creating this directory, and it isn't currently. Interestingly though, I do not think this was a problem before. Could it be that there was an application change that is now using this directory (with direct-upload disabled) where before it wasn't?
- Author Owner
A proposed fix for omnnibus is in gitlab-org/omnibus-gitlab!3396 (merged)
- Owner
I wonder if this is caused by https://dev.gitlab.org/gitlab/gitlabhq/issues/2860
- Author Owner
Rolling back gitlab-ee on preprod to 12.0.2 to see if we didn't see the problem with the previous release.
- Maintainer
@jarv the problem you have is described in https://gitlab.com/gitlab-org/gitlab-ce/issues/47220
- John Jarvis mentioned in merge request gitlab-org/omnibus-gitlab!3396 (merged)
mentioned in merge request gitlab-org/omnibus-gitlab!3396 (merged)
- Author Owner
@marin uploads are now working on preprod by ensuring manually that the uploads directory exists. I do not believe the problems here have anything to do with the new release, as we saw the issue after rolling back as well.
I will keep this open until gitlab-org/omnibus-gitlab!3396 (merged) is resolved since this the main reason for the failure.
- DJ Mountney closed via merge request gitlab-org/omnibus-gitlab!3396 (merged)
closed via merge request gitlab-org/omnibus-gitlab!3396 (merged)
- DJ Mountney mentioned in commit gitlab-org/omnibus-gitlab@d73ca7a1
mentioned in commit gitlab-org/omnibus-gitlab@d73ca7a1
- Marin Jankovski added teamDelivery + 1 deleted label
added teamDelivery + 1 deleted label
- Marin Jankovski added DeliveryP2 label and removed @gitlab-org~3857523 label
added DeliveryP2 label and removed @gitlab-org~3857523 label