<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary After upgrading from GitLab 12.8.7-ce to version 12.9.0-ce uploading job artifacts fails with this error. This issue occurs on a self-hosted Omnibus install, which is behind an NGINX reverse proxy. The issue happens with a newly setup runner (so it doesn't seem to be related to invalid tokens). ```bash WARNING: Uploading artifacts to coordinator... failed id=79119 responseStatus=500 Internal Server Error status=500 Internal Server Error token=ujxBxoBE WARNING: Retrying... context=artifacts-uploader error=invalid argument WARNING: Uploading artifacts to coordinator... failed id=79119 responseStatus=500 Internal Server Error status=500 Internal Server Error token=ujxBxoBE WARNING: Retrying... context=artifacts-uploader error=invalid argument WARNING: Uploading artifacts to coordinator... failed id=79119 responseStatus=500 Internal Server Error status=500 Internal Server Error token=ujxBxoBE FATAL: invalid argument ERROR: Job failed: exit status 1 ``` It seems that only the upload of artifacts is affected, e.g. attachments to issues can still be uploaded via the Web UI. There is an error and an exception stack trace in the production log (see full stack trace below): ``` Can't verify CSRF token authenticity. This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection` Unlike the logs may suggest, this does not result in an actual 422 response to the user For API requests, the only effect is that `current_user` will be `nil` for the duration of the request [...] Errno::EINVAL (Invalid argument @ rb_sysopen - /media/gitlab_artifacts/tmp/work/1585255026-17233-0002-3177/artifacts.zip): /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `initialize' /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `open' /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `mime_magic_content_type' /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:29:in `check_content_type_whitelist!' ``` I didn't see any errors in syslog or in the other log files created by GitLab. If there are any options to increase the verbosity of theses logs, please let me know. The uploaded artifact file mentioned in the production.log below does not exist on disk (`/media/gitlab_artifacts/tmp/work/1585255026-17233-0002-3177/artifacts.zip`). Artifacts are configured to be stored on a mounted network location. The location is mounted and the folder has the following permissions: ```bash ll /media/gitlab_artifacts/ total 36 drwx------ 2 git git 32768 Feb 4 14:23 ./ drwxr-xr-x 8 root root 4096 Jan 15 15:33 ../ drwx------ 2 git git 0 Mar 21 2018 44/ drwx------ 2 git git 0 Sep 17 2019 48/ drwx------ 2 git git 0 May 24 2018 4a/ drwx------ 2 git git 0 Sep 17 2019 4e/ drwx------ 2 git git 0 Jun 24 2018 59/ drwx------ 2 git git 0 Feb 14 08:08 95/ drwx------ 2 git git 0 Mar 16 2018 d5/ drwx------ 2 git git 0 Mar 6 2018 eb/ drwx------ 2 git git 0 Aug 29 2019 ff/ drwx------ 2 git git 0 Mar 16 2018 tmp/ ``` ### .gitlab-ci.yml For demo purposes, and to exclude any issues with the artifact created, I use this very basic pipeline: ```yaml test: script: date > date.txt artifacts: paths: - date.txt expire_in: 1 week ``` ### Relevant logs NGINX access.log (reverse proxy) ``` 10.192.xxx.xxx - - [26/Mar/2020:21:37:06 +0100] "POST /gitlab/api/v4/jobs/79119/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+week HTTP/1.1" 500 39 "-" "gitlab-runner 12.9.0 (12-9-stable; go1.13.8; windows/amd64)" ``` Exception in production.log: ``` Started GET "/gitlab/api/v4/internal/authorized_keys?key=[FILTERED]" for 127.0.0.1 at 2020-03-26 21:37:04 +0100 Started GET "/gitlab/api/v4/internal/authorized_keys?key=[FILTERED]" for 127.0.0.1 at 2020-03-26 21:37:04 +0100 Processing by Projects::PipelinesController#index as JSON Parameters: {"scope"=>"all", "page"=>"1", "namespace_id"=>"testuser", "project_id"=>"test-project"} Started GET "/gitlab/api/v4/internal/authorized_keys?key=[FILTERED]" for 127.0.0.1 at 2020-03-26 21:37:05 +0100 Started POST "/gitlab/api/v4/internal/allowed" for 127.0.0.1 at 2020-03-26 21:37:05 +0100 Started POST "/gitlab/api/v4/jobs/79119/artifacts/authorize?artifact_format=zip&artifact_type=archive&expire_in=1+week" for 127.0.0.1 at 2020-03-26 21:37:05 +0100 Started POST "/gitlab/api/v4/internal/allowed" for 127.0.0.1 at 2020-03-26 21:37:06 +0100 Completed 200 OK in 874ms (Views: 2.7ms | ActiveRecord: 121.4ms | Elasticsearch: 0.0ms | Allocations: 295422) Started POST "/gitlab/api/v4/jobs/79119/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+week" for 127.0.0.1 at 2020-03-26 21:37:06 +0100 Processing by UploadsController#show as HTML Parameters: {"width"=>"26", "model"=>"user", "mounted_as"=>"avatar", "id"=>"2", "filename"=>"avatar.jpg"} Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML Parameters: {"file.sha256"=>"960a30e6dba05045b608c52dbd3e12ab2a571a3626ab15522e075dc21302407e", "file.sha512"=>"40a5c676f6ea71166682eb591d2c55dd1f95673cce8c120fafcc26ae5325242e9f e55d8be15d5b35cf037c9622789a33a51f4c61f9f4893bce405bf8b3b7f4c1", "file.name"=>"artifacts.zip", "file.path"=>"/media/gitlab_artifacts/tmp/uploads/artifacts.zip419031322", "file.size "=>"242", "file.md5"=>"3204a4418f2d7eb4e65ba763f441841e", "file.sha1"=>"0312ccf89edb1f61b894cd6ba536a493250ef9b6", "metadata.sha512"=>"713d49982c350fde574c2755ebb1bd2a50a3fef6286b7 bb77cfd7856394bacc8ff024aee714c657ff3d009e16af9c2ff9f6d7051ba5aff4969c753d7f69ecd6d", "metadata.name"=>"metadata.gz", "metadata.path"=>"/media/gitlab_artifacts/tmp/uploads/metadata .gz990188977", "metadata.size"=>"152", "metadata.md5"=>"fa175dba886bce3ae5793411aff5f407", "metadata.sha1"=>"d948907fb69570e29fdbd320e6e2289cf8d9eda1", "metadata.sha256"=>"93312c6e 55f2c704440ffb119899de7eba69b441e7f52836b28e3ca17967e721", "artifact_format"=>"zip", "artifact_type"=>"archive", "expire_in"=>"1 week"} Can't verify CSRF token authenticity. This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection` Unlike the logs may suggest, this does not result in an actual 422 response to the user For API requests, the only effect is that `current_user` will be `nil` for the duration of the request Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 255) Processing by UploadsController#show as HTML Parameters: {"width"=>"20", "model"=>"user", "mounted_as"=>"avatar", "id"=>"2", "filename"=>"avatar.jpg"} Errno::EINVAL (Invalid argument @ rb_sysopen - /media/gitlab_artifacts/tmp/work/1585255026-17233-0002-3177/artifacts.zip): /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `initialize' /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `open' /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:45:in `mime_magic_content_type' /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/content_type_whitelist.rb:29:in `check_content_type_whitelist!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/callbacks.rb:14:in `block in with_callbacks' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/callbacks.rb:14:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/callbacks.rb:14:in `with_callbacks' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/uploader/cache.rb:137:in `cache!' /opt/gitlab/embedded/service/gitlab-rails/app/uploaders/object_storage.rb:325:in `cache!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:44:in `block in cache' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:42:in `map' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:42:in `cache' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mount.rb:146:in `file=' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/orm/activerecord.rb:70:in `file=' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:51:in `public_send' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:51:in `_assign_attribute' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:44:in `block in _assign_attributes' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:43:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:43:in `_assign_attributes' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/attribute_assignment.rb:22:in `_assign_attributes' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activemodel-6.0.2/lib/active_model/attribute_assignment.rb:35:in `assign_attributes' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/core.rb:326:in `initialize' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/inheritance.rb:70:in `new' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activerecord-6.0.2/lib/active_record/inheritance.rb:70:in `new' /opt/gitlab/embedded/service/gitlab-rails/app/services/ci/create_job_artifacts_service.rb:29:in `build_artifact' /opt/gitlab/embedded/service/gitlab-rails/app/services/ci/create_job_artifacts_service.rb:15:in `execute' /opt/gitlab/embedded/service/gitlab-rails/lib/api/runner.rb:286:in `block (2 levels) in <class:Runner>' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in `instrument' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/notifications.rb:182:in `instrument' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in `_call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `catch' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/mapper.rb:48:in `serve' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:49:in `block in serve' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:32:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/journey/router.rb:32:in `serve' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/routing/route_set.rb:837:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/rails_queue_duration.rb:27:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `block in call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:62:in `run' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/rack_middleware.rb:17:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:17:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.11.0/lib/labkit/middleware/rack.rb:19:in `block in call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.11.0/lib/labkit/context.rb:31:in `with_context' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.11.0/lib/labkit/middleware/rack.rb:18:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:121:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-6.2.0/lib/rack/attack.rb:169:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/http/content_security_policy.rb:18:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:53:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/cookies.rb:648:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/callbacks.rb:101:in `run_callbacks' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/callbacks.rb:26:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:38:in `call_app' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:26:in `block in call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in `block in tagged' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:28:in `tagged' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-6.0.2/lib/active_support/tagged_logging.rb:80:in `tagged' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/rack/logger.rb:26:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/remote_ip.rb:81:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/request_context.rb:23:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/request_id.rb:27:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/fix_local_cache_middleware.rb:9:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/executor.rb:14:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/lock.rb:17:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-6.0.2/lib/action_dispatch/middleware/host_authorization.rb:77:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/engine.rb:526:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `public_send' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-6.0.2/lib/rails/railtie.rb:190:in `method_missing' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>' /opt/gitlab/embedded/bin/unicorn:23:in `load' /opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors' /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' ``` ### Output of checks (If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com) #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> gitlab-rake gitlab:env:info System information System: Ubuntu 18.04 Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.9.0 Revision: 9a382ff2c82 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.12 URL: https://gitlab.example.com/gitlab HTTP Clone URL: https://gitlab.example.com/gitlab/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 12.0.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 12.0.0 ? ... OK (12.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... E0326 22:11:21.880721213 8894 http_proxy.cc:64] 'https' scheme not supported in proxy URI default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit. Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/1 ... yes 6/3 ... yes 5/6 ... yes [...] 5/307 ... yes 5/308 ... yes 61/309 ... yes 111/310 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 102 Is authorized keys file accessible? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished </pre> </details>