Artifact upload broken after upgrade to GitLab 12.9: FATAL: invalid argument
Summary
After upgrading from GitLab 12.8.7-ce to version 12.9.0-ce uploading job artifacts fails with this error. This issue occurs on a self-hosted Omnibus install, which is behind an NGINX reverse proxy.
The issue happens with a newly setup runner (so it doesn't seem to be related to invalid tokens).
WARNING: Uploading artifacts to coordinator... failed id=79119 responseStatus=500 Internal Server Error status=500 Internal Server Error token=ujxBxoBE
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts to coordinator... failed id=79119 responseStatus=500 Internal Server Error status=500 Internal Server Error token=ujxBxoBE
WARNING: Retrying... context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts to coordinator... failed id=79119 responseStatus=500 Internal Server Error status=500 Internal Server Error token=ujxBxoBE
FATAL: invalid argument
ERROR: Job failed: exit status 1It seems that only the upload of artifacts is affected, e.g. attachments to issues can still be uploaded via the Web UI.
There is an error and an exception stack trace in the production log (see full stack trace below):
Can't verify CSRF token authenticity.
This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that `current_user` will be `nil` for the duration of the request
[...]
Errno::EINVAL (Invalid argument @ rb_sysopen - /media/gitlab_artifacts/tmp/work/1585255026-17233-0002-3177/artifacts.zip):
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `open'
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `mime_magic_content_type'
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:29:in `check_content_type_whitelist!'I didn't see any errors in syslog or in the other log files created by GitLab. If there are any options to increase the verbosity of theses logs, please let me know.
The uploaded artifact file mentioned in the production.log below does not exist on disk (/media/gitlab_artifacts/tmp/work/1585255026-17233-0002-3177/artifacts.zip).
Artifacts are configured to be stored on a mounted network location. The location is mounted and the folder has the following permissions:
ll /media/gitlab_artifacts/
total 36
drwx------ 2 git git 32768 Feb 4 14:23 ./
drwxr-xr-x 8 root root 4096 Jan 15 15:33 ../
drwx------ 2 git git 0 Mar 21 2018 44/
drwx------ 2 git git 0 Sep 17 2019 48/
drwx------ 2 git git 0 May 24 2018 4a/
drwx------ 2 git git 0 Sep 17 2019 4e/
drwx------ 2 git git 0 Jun 24 2018 59/
drwx------ 2 git git 0 Feb 14 08:08 95/
drwx------ 2 git git 0 Mar 16 2018 d5/
drwx------ 2 git git 0 Mar 6 2018 eb/
drwx------ 2 git git 0 Aug 29 2019 ff/
drwx------ 2 git git 0 Mar 16 2018 tmp/.gitlab-ci.yml
For demo purposes, and to exclude any issues with the artifact created, I use this very basic pipeline:
test:
script: date > date.txt
artifacts:
paths:
- date.txt
expire_in: 1 weekRelevant logs
NGINX access.log (reverse proxy)
10.192.xxx.xxx - - [26/Mar/2020:21:37:06 +0100] "POST /gitlab/api/v4/jobs/79119/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+week HTTP/1.1" 500 39 "-" "gitlab-runner 12.9.0 (12-9-stable; go1.13.8; windows/amd64)"Exception in production.log:
Started GET "/gitlab/api/v4/internal/authorized_keys?key=[FILTERED]" for 127.0.0.1 at 2020-03-26 21:37:04 +0100
Started GET "/gitlab/api/v4/internal/authorized_keys?key=[FILTERED]" for 127.0.0.1 at 2020-03-26 21:37:04 +0100
Processing by Projects::PipelinesController#index as JSON
Parameters: {"scope"=>"all", "page"=>"1", "namespace_id"=>"testuser", "project_id"=>"test-project"}
Started GET "/gitlab/api/v4/internal/authorized_keys?key=[FILTERED]" for 127.0.0.1 at 2020-03-26 21:37:05 +0100
Started POST "/gitlab/api/v4/internal/allowed" for 127.0.0.1 at 2020-03-26 21:37:05 +0100
Started POST "/gitlab/api/v4/jobs/79119/artifacts/authorize?artifact_format=zip&artifact_type=archive&expire_in=1+week" for 127.0.0.1 at 2020-03-26 21:37:05 +0100
Started POST "/gitlab/api/v4/internal/allowed" for 127.0.0.1 at 2020-03-26 21:37:06 +0100
Completed 200 OK in 874ms (Views: 2.7ms | ActiveRecord: 121.4ms | Elasticsearch: 0.0ms | Allocations: 295422)
Started POST "/gitlab/api/v4/jobs/79119/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+week" for 127.0.0.1 at 2020-03-26 21:37:06 +0100
Processing by UploadsController#show as HTML
Parameters: {"width"=>"26", "model"=>"user", "mounted_as"=>"avatar", "id"=>"2", "filename"=>"avatar.jpg"}
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Parameters: {"file.sha256"=>"960a30e6dba05045b608c52dbd3e12ab2a571a3626ab15522e075dc21302407e", "file.sha512"=>"40a5c676f6ea71166682eb591d2c55dd1f95673cce8c120fafcc26ae5325242e9f
e55d8be15d5b35cf037c9622789a33a51f4c61f9f4893bce405bf8b3b7f4c1", "file.name"=>"artifacts.zip", "file.path"=>"/media/gitlab_artifacts/tmp/uploads/artifacts.zip419031322", "file.size
"=>"242", "file.md5"=>"3204a4418f2d7eb4e65ba763f441841e", "file.sha1"=>"0312ccf89edb1f61b894cd6ba536a493250ef9b6", "metadata.sha512"=>"713d49982c350fde574c2755ebb1bd2a50a3fef6286b7
bb77cfd7856394bacc8ff024aee714c657ff3d009e16af9c2ff9f6d7051ba5aff4969c753d7f69ecd6d", "metadata.name"=>"metadata.gz", "metadata.path"=>"/media/gitlab_artifacts/tmp/uploads/metadata
.gz990188977", "metadata.size"=>"152", "metadata.md5"=>"fa175dba886bce3ae5793411aff5f407", "metadata.sha1"=>"d948907fb69570e29fdbd320e6e2289cf8d9eda1", "metadata.sha256"=>"93312c6e
55f2c704440ffb119899de7eba69b441e7f52836b28e3ca17967e721", "artifact_format"=>"zip", "artifact_type"=>"archive", "expire_in"=>"1 week"}
Can't verify CSRF token authenticity.
This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that `current_user` will be `nil` for the duration of the request
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 255)
Processing by UploadsController#show as HTML
Parameters: {"width"=>"20", "model"=>"user", "mounted_as"=>"avatar", "id"=>"2", "filename"=>"avatar.jpg"}
Errno::EINVAL (Invalid argument @ rb_sysopen - /media/gitlab_artifacts/tmp/work/1585255026-17233-0002-3177/artifacts.zip):
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `initialize'
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `open'
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `mime_magic_content_type'
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:29:in `check_content_type_whitelist!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/callbacks.rb:14:in `block in with_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/callbacks.rb:14:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/callbacks.rb:14:in `with_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/cache.rb:137:in `cache!'
/opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:325:in `cache!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:44:in `block in cache'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:42:in `map'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:42:in `cache'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mount.rb:146:in `file='
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/orm/activerecord.rb:70:in `file='
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:51:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:51:in `_assign_attribute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:44:in `block in _assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:43:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:43:in `_assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/attribute_assignment.rb:22:in `_assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/core.rb:326:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/inheritance.rb:70:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/inheritance.rb:70:in `new'
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/create_job_artifacts_service.rb:29:in `build_artifact'
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/create_job_artifacts_service.rb:15:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/runner.rb:286:in `block (2 levels) in <class:Runner>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in `instrument'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:32:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:32:in `serve'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/route_set.rb:837:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:62:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.11.0/lib/labkit/middleware/rack.rb:19:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.11.0/lib/labkit/context.rb:31:in `with_context'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.11.0/lib/labkit/middleware/rack.rb:18:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:121:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-6.2.0/lib/rack/attack.rb:169:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:53:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/cookies.rb:648:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:101:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:38:in `call_app'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:26:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in `block in tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:28:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in `tagged'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:26:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:23:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/lock.rb:17:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/host_authorization.rb:77:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:526:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `public_send'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:23:in `load'
/opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
gitlab-rake gitlab:env:info System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.9.0 Revision: 9a382ff2c82 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.12 URL: https://gitlab.example.com/gitlab HTTP Clone URL: https://gitlab.example.com/gitlab/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 12.0.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.0.0 ? ... OK (12.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... E0326 22:11:21.880721213 8894 http_proxy.cc:64] 'https' scheme not supported in proxy URI default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/1 ... yes 6/3 ... yes 5/6 ... yes [...] 5/307 ... yes 5/308 ... yes 61/309 ... yes 111/310 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 102 Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Edited by Dirk