External Object Storage Results in 500 internal server error for artifacts
Summary
I currently experience problems with an external instance of minio. Everything works, the only thing that does not is uploading artifacts. In the runner it will return a 500
. In the unicorn log (see below) a weird error "No address associated with hostname" is being thrown. I tried to dig a bit deeper, but lack time currently, so not sure if this is a bug or just a problem on my side. However the configuration is correct and the host can actually resolve the hostname given in the config for the artifacts. And everything was configured accordingly to the docs, so I doubt it is on my side, at least as of now :)
NOTE: Solution found in this comment. The external object storage did not support DNS lookup buckets, and path_style: true
needed to be set. Documentation can be added to doc/advanced/external-object-storage
& examples/objectstorage/*s3.yaml
for improved experience.
Versions
- Chart: latest stable (tagged version | branch | hash
git rev-parse HEAD
) - Platform:
- Cloud: (GKE | AKS | EKS | ?)
- Self-hosted: Rancher RKE
- Kubernetes: (
kubectl version
)- Client:
- Server:
- Helm: (
helm version
)- Client:
- Server:
Relevant logs
Excon::Error::Socket (getaddrinfo: No address associated with hostname (SocketError)):
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/socket.rb:102:in `getaddrinfo'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/socket.rb:102:in `connect'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/ssl_socket.rb:155:in `connect'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/socket.rb:30:in `initialize'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/ssl_socket.rb:10:in `initialize'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/connection.rb:429:in `new'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/connection.rb:429:in `socket'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/connection.rb:111:in `request_call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/middlewares/mock.rb:48:in `request_call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/middlewares/instrumentor.rb:26:in `request_call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/middlewares/base.rb:16:in `request_call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/excon-0.62.0/lib/excon/connection.rb:260:in `request'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:35:in `request'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/fog-aws-3.3.0/lib/fog/aws/storage.rb:614:in `_request'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/fog-aws-3.3.0/lib/fog/aws/storage.rb:609:in `request'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/fog-aws-3.3.0/lib/fog/aws/requests/storage/initiate_multipart_upload.rb:29:in `initiate_multipart_upload'
/srv/gitlab/lib/object_storage/direct_upload.rb:151:in `block in upload_id'
/srv/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/srv/gitlab/lib/object_storage/direct_upload.rb:150:in `upload_id'
/srv/gitlab/lib/object_storage/direct_upload.rb:94:in `multipart_part_upload_url'
/srv/gitlab/lib/object_storage/direct_upload.rb:84:in `block in multipart_part_urls'
/srv/gitlab/lib/object_storage/direct_upload.rb:83:in `initialize'
/srv/gitlab/lib/object_storage/direct_upload.rb:83:in `new'
/srv/gitlab/lib/object_storage/direct_upload.rb:83:in `multipart_part_urls'
/srv/gitlab/lib/object_storage/direct_upload.rb:57:in `multipart_upload_hash'
/srv/gitlab/lib/object_storage/direct_upload.rb:46:in `to_hash'
/srv/gitlab/app/uploaders/object_storage.rb:202:in `workhorse_remote_upload_options'
/srv/gitlab/app/uploaders/object_storage.rb:184:in `workhorse_authorize'
/srv/gitlab/lib/api/runner.rb:228:in `block (2 levels) in <class:Runner>'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:166:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:166:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:17:in `block in <class:Constraints>'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:46:in `serve'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:39:in `block in serve'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `each'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `serve'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/route_set.rb:727:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
/srv/gitlab/lib/gitlab/middleware/multipart.rb:103:in `call'
/srv/gitlab/lib/gitlab/request_profiler/middleware.rb:16:in `call'
/srv/gitlab/ee/lib/gitlab/jira/middleware.rb:17:in `call'
/srv/gitlab/lib/gitlab/middleware/go.rb:20:in `call'
/srv/gitlab/lib/gitlab/etag_caching/middleware.rb:13:in `call'
/srv/gitlab/lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
/srv/gitlab/lib/gitlab/correlation_id.rb:15:in `use_id'
/srv/gitlab/lib/gitlab/middleware/correlation_id.rb:15:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/batch-loader-1.2.2/lib/batch_loader/middleware.rb:11:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/srv/gitlab/lib/gitlab/middleware/read_only/controller.rb:42:in `call'
/srv/gitlab/lib/gitlab/middleware/read_only.rb:18:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/cookies.rb:613:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:90:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/srv/gitlab/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:36:in `call_app'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in `block in tagged'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:26:in `tagged'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in `tagged'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in `call'
/srv/gitlab/lib/gitlab/request_context.rb:26:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/request_id.rb:24:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/srv/gitlab/lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/sentry-raven-2.7.4/lib/raven/integrations/rack.rb:51:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/engine.rb:522:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in `public_send'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in `method_missing'
/srv/gitlab/lib/gitlab/middleware/release_env.rb:13:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in `start'
/srv/gitlab/vendor/bundle/ruby/2.5.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `load'
/srv/gitlab/vendor/bundle/ruby/2.5.0/bin/unicorn:23:in `<top (required)>'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'