<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary <!-- Summarize the bug encountered concisely. --> Some S3-compatible object storage backends don't conform strictly to [RFC 9110](https://datatracker.ietf.org/doc/html/rfc9110#name-content-range) specification for `Content-Range` header. In this scenario, Ruby's `net-http` returns a `Net::HTTPHeaderSyntaxError` error, and GitLab does not handle this. The result is the data is not returned from object storage at all. A customer is seeing HammerspaceS3 respond with status code `206` and `"content-range"=>["0-16366/16367"]`, resulting in exception `Net::HTTPHeaderSyntaxError`. It doesn't match the regex that net-http uses to parse the content range: https://github.com/ruby/net-http/blob/87cf61d2ad8a331ee206b7d142d38b7979396c98/lib/net/http/header.rb#L718 ### Steps to reproduce <!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. --> 1. Use an S3-compatible object storage backend for job artifacts (HammerspaceS3) 2. Run pipelines and ensure the jobs complete 3. Attempt to view archived job traces via UI or API ### Example Project <!-- If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report. If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version. --> ### What is the current *bug* behavior? <!-- Describe what actually happens. --> A `500` status code error occurs with exception `Net::HTTPHeaderSyntaxError` ### What is the expected *correct* behavior? <!-- Describe what you should see instead. --> GitLab handles this error gracefully ### Relevant logs and/or screenshots <!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise. --> ``` "exception.class": "Net::HTTPHeaderSyntaxError", "exception.message": "wrong Content-Range format", "exception.backtrace": [ "net-http (0.6.0) lib/net/http/header.rb:673:in `content_range'", "lib/gitlab/http_io.rb:164:in `get_chunk'", "lib/gitlab/http_io.rb:114:in `readline'", "lib/gitlab/http_io.rb:71:in `each_line'", "lib/gitlab/ci/ansi2json/converter.rb:71:in `process_stream_with_lookahead'", "lib/gitlab/ci/ansi2json/converter.rb:38:in `convert'", "lib/gitlab/ci/ansi2json.rb:8:in `convert'", "app/models/ci/build_trace.rb:15:in `initialize'", "app/controllers/projects/jobs_controller.rb:77:in `new'", "app/controllers/projects/jobs_controller.rb:77:in `block (3 levels) in trace'", "actionpack (7.2.3.1) lib/action_controller/metal/mime_responds.rb:224:in `respond_to'", "app/controllers/projects/jobs_controller.rb:75:in `block in trace'", "lib/gitlab/ci/trace.rb:172:in `read_stream'", "lib/gitlab/ci/trace.rb:94:in `read'", "app/controllers/projects/jobs_controller.rb:74:in `trace'", "actionpack (7.2.3.1) lib/action_controller/metal/basic_implicit_render.rb:8:in `send_action'", "actionpack (7.2.3.1) lib/abstract_controller/base.rb:215:in `process_action'", "actionpack (7.2.3.1) lib/action_controller/metal/rendering.rb:193:in `process_action'", "actionpack (7.2.3.1) lib/abstract_controller/callbacks.rb:261:in `block in process_action'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:121:in `block in run_callbacks'", "app/controllers/application_controller.rb:516:in `set_current_admin'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'", "lib/gitlab/session.rb:11:in `with_session'", "app/controllers/application_controller.rb:507:in `set_session_storage'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'", "lib/gitlab/i18n.rb:116:in `with_locale'", "lib/gitlab/i18n.rb:122:in `with_user_locale'", "app/controllers/application_controller.rb:498:in `set_locale'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'", "app/controllers/application_controller.rb:487:in `set_current_context'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'", "lib/gitlab/ip_address_state.rb:11:in `with'", "app/controllers/application_controller.rb:493:in `set_current_ip_address'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'", "marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'", "sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:34:in `block in sentry_around_action'", "sentry-ruby (5.23.0) lib/sentry/hub.rb:138:in `with_child_span'", "sentry-ruby (5.23.0) lib/sentry-ruby.rb:515:in `with_child_span'", "sentry-rails (5.23.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:130:in `block in run_callbacks'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:141:in `run_callbacks'", "actionpack (7.2.3.1) lib/abstract_controller/callbacks.rb:260:in `process_action'", "actionpack (7.2.3.1) lib/action_controller/metal/rescue.rb:27:in `process_action'", "actionpack (7.2.3.1) lib/action_controller/metal/instrumentation.rb:77:in `block in process_action'", "activesupport (7.2.3.1) lib/active_support/notifications.rb:210:in `block in instrument'", "activesupport (7.2.3.1) lib/active_support/notifications/instrumenter.rb:58:in `instrument'", "activesupport (7.2.3.1) lib/active_support/notifications.rb:210:in `instrument'", "actionpack (7.2.3.1) lib/action_controller/metal/instrumentation.rb:76:in `process_action'", "actionpack (7.2.3.1) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'", "activerecord (7.2.3.1) lib/active_record/railties/controller_runtime.rb:39:in `process_action'", "actionpack (7.2.3.1) lib/abstract_controller/base.rb:152:in `process'", "actionview (7.2.3.1) lib/action_view/rendering.rb:40:in `process'", "actionpack (7.2.3.1) lib/action_controller/metal.rb:252:in `dispatch'", "actionpack (7.2.3.1) lib/action_controller/metal.rb:333:in `block in dispatch'", "lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'", "actionpack (7.2.3.1) lib/action_controller/metal.rb:333:in `dispatch'", "actionpack (7.2.3.1) lib/action_dispatch/routing/route_set.rb:67:in `dispatch'", "actionpack (7.2.3.1) lib/action_dispatch/routing/route_set.rb:50:in `serve'", "actionpack (7.2.3.1) lib/action_dispatch/routing/mapper.rb:32:in `block in <class:Constraints>'", "actionpack (7.2.3.1) lib/action_dispatch/routing/mapper.rb:62:in `serve'", "actionpack (7.2.3.1) lib/action_dispatch/journey/router.rb:53:in `block in serve'", "config/initializers/action_dispatch_journey_router.rb:52:in `block in find_routes'", "config/initializers/action_dispatch_journey_router.rb:25:in `map!'", "config/initializers/action_dispatch_journey_router.rb:25:in `find_routes'", "actionpack (7.2.3.1) lib/action_dispatch/journey/router.rb:34:in `serve'", "actionpack (7.2.3.1) lib/action_dispatch/routing/route_set.rb:896:in `call'", "gitlab-experiment (1.6.0) lib/gitlab/experiment/middleware.rb:19:in `call'", "flipper (1.3.6) lib/flipper/middleware/memoizer.rb:87:in `memoized_call'", "flipper (1.3.6) lib/flipper/middleware/memoizer.rb:45:in `call'", "omniauth (2.1.4) lib/omniauth/strategy.rb:202:in `call!'", "omniauth (2.1.4) lib/omniauth/strategy.rb:169:in `call'", "lib/gitlab/metrics/zoekt_rack_middleware.rb:16:in `call'", "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'", "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'", "lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'", "lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'", "lib/gitlab/middleware/memory_report.rb:13:in `call'", "lib/gitlab/middleware/speedscope.rb:13:in `call'", "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'", "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'", "lib/gitlab/etag_caching/middleware.rb:21:in `call'", "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'", "lib/gitlab/metrics/web_transaction.rb:46:in `run'", "lib/gitlab/metrics/rack_middleware.rb:16:in `call'", "lib/gitlab/middleware/go.rb:21:in `call'", "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'", "lib/gitlab/database/query_analyzer.rb:94:in `within'", "lib/gitlab/middleware/query_analyzer.rb:11:in `call'", "lib/ci/job_token/middleware.rb:11:in `call'", "batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'", "rack-attack (6.8.0) lib/rack/attack.rb:105:in `call'", "apollo_upload_server (2.1.8) lib/apollo_upload_server/middleware.rb:14:in `call'", "lib/gitlab/middleware/multipart.rb:177:in `call'", "lib/gitlab/middleware/rack_attack_headers.rb:42:in `call'", "rack-attack (6.8.0) lib/rack/attack.rb:129:in `call'", "lib/gitlab/middleware/labkit_rack_rate_limit.rb:45:in `call'", "warden (1.2.9) lib/warden/manager.rb:36:in `block in call'", "warden (1.2.9) lib/warden/manager.rb:34:in `catch'", "warden (1.2.9) lib/warden/manager.rb:34:in `call'", "rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'", "rack (2.2.23) lib/rack/tempfile_reaper.rb:15:in `call'", "rack (2.2.23) lib/rack/etag.rb:27:in `call'", "rack (2.2.23) lib/rack/conditional_get.rb:27:in `call'", "rack (2.2.23) lib/rack/head.rb:12:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/http/permissions_policy.rb:38:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/http/content_security_policy.rb:38:in `call'", "lib/gitlab/middleware/read_only/controller.rb:40:in `call'", "lib/gitlab/middleware/read_only.rb:18:in `call'", "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'", "rack (2.2.23) lib/rack/session/abstract/id.rb:266:in `context'", "rack (2.2.23) lib/rack/session/abstract/id.rb:260:in `call'", "lib/gitlab/middleware/secure_headers.rb:11:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/cookies.rb:704:in `call'", "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/callbacks.rb:31:in `block in call'", "activesupport (7.2.3.1) lib/active_support/callbacks.rb:101:in `run_callbacks'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/callbacks.rb:30:in `call'", "sentry-rails (5.23.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/debug_exceptions.rb:31:in `call'", "lib/gitlab/middleware/path_depth_check.rb:32:in `call'", "lib/gitlab/middleware/path_traversal_check.rb:40:in `call'", "lib/gitlab/middleware/handle_malformed_strings.rb:19:in `call'", "lib/gitlab/middleware/json_validation.rb:233:in `call'", "sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'", "sentry-ruby (5.23.0) lib/sentry/hub.rb:299:in `with_session_tracking'", "sentry-ruby (5.23.0) lib/sentry-ruby.rb:428:in `with_session_tracking'", "sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'", "sentry-ruby (5.23.0) lib/sentry/hub.rb:89:in `with_scope'", "sentry-ruby (5.23.0) lib/sentry-ruby.rb:408:in `with_scope'", "sentry-ruby (5.23.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/show_exceptions.rb:32:in `call'", "lib/gitlab/middleware/basic_health_check.rb:25:in `call'", "lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'", "railties (7.2.3.1) lib/rails/rack/logger.rb:29:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/remote_ip.rb:96:in `call'", "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'", "lib/gitlab/middleware/request_context.rb:15:in `call'", "lib/gitlab/middleware/duo_workflow_id.rb:16:in `call'", "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'", "request_store (1.7.0) lib/request_store/middleware.rb:19:in `call'", "rack (2.2.23) lib/rack/method_override.rb:24:in `call'", "rack (2.2.23) lib/rack/runtime.rb:22:in `call'", "rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'", "rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'", "rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'", "config/initializers/fix_local_cache_middleware.rb:11:in `call'", "lib/gitlab/middleware/compressed_json.rb:44:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/executor.rb:16:in `call'", "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'", "rack (2.2.23) lib/rack/sendfile.rb:127:in `call'", "lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'", "lib/gitlab/middleware/request_cost.rb:16:in `call'", "gitlab-labkit (2.7.0) lib/labkit/middleware/rack.rb:22:in `block in call'", "gitlab-labkit (2.7.0) lib/labkit/context.rb:43:in `with_context'", "gitlab-labkit (2.7.0) lib/labkit/middleware/rack.rb:21:in `call'", "actionpack (7.2.3.1) lib/action_dispatch/middleware/request_id.rb:33:in `call'", "lib/gitlab/middleware/static_assets_authorization.rb:28:in `call'", "railties (7.2.3.1) lib/rails/engine.rb:535:in `call'", "railties (7.2.3.1) lib/rails/railtie.rb:226:in `public_send'", "railties (7.2.3.1) lib/rails/railtie.rb:226:in `method_missing'", "lib/gitlab/middleware/release_env.rb:12:in `call'", "rack (2.2.23) lib/rack/urlmap.rb:74:in `block in call'", "rack (2.2.23) lib/rack/urlmap.rb:58:in `each'", "rack (2.2.23) lib/rack/urlmap.rb:58:in `call'", "puma (8.0.2) lib/puma/configuration.rb:305:in `call'", "puma (8.0.2) lib/puma/response.rb:79:in `block in handle_request'", "puma (8.0.2) lib/puma/thread_pool.rb:434:in `with_force_shutdown'", "puma (8.0.2) lib/puma/response.rb:78:in `handle_request'", "puma (8.0.2) lib/puma/server.rb:508:in `process_client'", "puma (8.0.2) lib/puma/server.rb:263:in `block in run'", "puma (8.0.2) lib/puma/thread_pool.rb:246:in `block in spawn_thread'" ], ``` ### Output of checks <!-- If you are reporting a bug on GitLab.com, uncomment below --> <!-- This bug happens on GitLab.com --> <!-- and uncomment below if you have /label privileges --> <!-- /label ~"reproduced on GitLab.com" --> <!-- or follow up with an issue comment of `@gitlab-bot label ~"reproduced on GitLab.com"` if you do not --> #### Results of GitLab environment info <!-- Input any relevant GitLab environment information if needed. --> <details> <summary>Expand for output related to GitLab environment info</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) </pre> </details> #### Results of GitLab application Check <!-- Input any relevant GitLab application check information if needed. --> <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes <!-- If you can, link to the line of code that might be responsible for the problem. --> ### Patch release information for backports If the bug fix needs to be backported in a [patch release](https://handbook.gitlab.com/handbook/engineering/releases/patch-releases) to a version under [the maintenance policy](https://docs.gitlab.com/policy/maintenance/), please follow the steps on the [patch release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md). Refer to the [internal "Release Information" dashboard](https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1) for information about the next patch release, including the targeted versions, expected release date, and current status. #### High-severity bug remediation To remediate high-severity issues requiring an [internal release](https://handbook.gitlab.com/handbook/engineering/releases/internal-releases/) for single-tenant SaaS instances, refer to the [internal release process for engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md?ref_type=heads). <!-- If you don't have /label privileges, follow up with an issue comment of `@gitlab-bot label ~"type::bug"` -->