Direct transfer reports "Invalid content type" when the source returns an error, hiding the real HTTP status

Summary

When a direct transfer migration downloads a relation export and the source instance answers with an error (for example 404 with a JSON body), the destination fails the relation with:

Import::BulkImports::FileDownloadStrategy::ServiceError: Invalid content type

HttpFileDownloadStrategy#perform_download validates the response Content-Type before it checks the status code, so any non-200 response with a JSON (or HTML) body raises "Invalid content type". The raise(ServiceError, "File download error #{chunk.code}") unless chunk.code == 200 check is never reached for these responses:

app/services/import/bulk_imports/http_file_download_strategy.rb Lines 47 to 57
47 48 49 50 51 52 53 54 55 56 57 
            unless @remote_content_validated
              validate_content_type

              @remote_content_validated = true
            end

            bytes_downloaded += chunk.size

            validate_size!(bytes_downloaded)

            raise(ServiceError, "File download error #{chunk.code}") unless chunk.code == 200

The troubleshooting docs only attribute this error to traffic bypassing Workhorse, so users and Support look at reverse proxy and network configuration when the source is actually returning an error. An Import engineer noted the same in #432553 (comment 1683841876): "The problem with 'invalid content type' error is it's a generic one and the underlying reason can always be different".

Customer report: https://gitlab.zendesk.com/agent/tickets/766066 (internal). The source was GitLab 18.11 with bundled NGINX routing to Workhorse. The source's Workhorse log for the failed relation=repository download showed "status":404,"content_type":"application/json","written_bytes":27, which is {"message":"404 Not Found"}. That matches the response when Allow migrating GitLab groups and projects by direct transfer is disabled on the source (lib/api/project_export.rb) or when the export file is missing on disk (present_carrierwave_file!). The retry succeeded after direct transfer was enabled on the source.

Steps to reproduce

  1. On the source instance, make sure Admin > Settings > General > Import and export settings > Allow migrating GitLab groups and projects by direct transfer is off.

  2. Request a relation export download as the destination does. The source answers 404 with a JSON body:

    curl --silent --output /dev/null --write-out 'status=%{http_code} content_type=%{content_type} bytes=%{size_download}\n' \
  "https://source.example.com/api/v4/projects/<id>/export_relations/download?relation=repository&private_token=<token>"
# status=404 content_type=application/json bytes=27

  3. On the destination, download the same relation with the direct transfer download strategy (Rails console):

    cfg = Struct.new(:url, :access_token).new('https://source.example.com', '<token>')
entity = Object.new
def entity.relation_download_url_path(rel, _b = nil) = "/projects/<id>/export_relations/download?relation=#{rel}"
ctx = Struct.new(:configuration, :entity, :bulk_import_id, :extra) do
  def override_file_size_limit? = false
  def offline? = false
end.new(cfg, entity, 0, {})

strategy = Import::BulkImports::HttpFileDownloadStrategy.new(context: ctx, relative_url: entity.relation_download_url_path('repository'))
Dir.mktmpdir { |dir| strategy.download_file(File.join(dir, 'repository.tar.gz')) }

Reproduced on a development instance at 34ef0d2e280 (19.6.0-pre), with the destination and source on the same instance. Requests to the local network were allowed for that console session only. Result:

raised Import::BulkImports::FileDownloadStrategy::ServiceError: Invalid content type
response_code=404 content-type=application/json body={"message":"404 Not Found"}

app/services/import/bulk_imports/, lib/bulk_imports/clients/http.rb and lib/api/project_export.rb are unchanged between 34ef0d2e280 and master at 9fecaab7103.

Example Project

N/A (any project, source with direct transfer disabled).

What is the current bug behavior?

A non-200, non-retriable response from the source is reported as Invalid content type. The source's status code and message are only in the BulkImports::Logger warning (response_code, last_chunk_context), not in the import failure that users see in the UI or the bulk_imports/:id/entities API.

What is the expected correct behavior?

The failure names the source's response, for example File download error 404: {"message":"404 Not Found"}, so users can fix the source (enable direct transfer, check the export) instead of their network. "Invalid content type" is only raised for a 200 response with an unexpected content type.

Relevant logs and/or screenshots

Source Workhorse log for the failed download (customer instance, 18.11):

{"backend_id":"rails","content_type":"application/json","method":"GET","msg":"access","route":"^/api/","status":404,"uri":"/api/v4/projects/<id>/export_relations/download?relation=repository&page=1&per_page=30&private_token=[FILTERED]","user_agent":"Ruby","written_bytes":27}

Destination failure:

{ "relation": "repository", "exception_message": "Invalid content type", "exception_class": "Import::BulkImports::FileDownloadStrategy::ServiceError" }

Output of checks

This bug happens on GitLab.com (destination).

Possible fixes

In HttpFileDownloadStrategy#perform_download, check the status code before validate_content_type, and include the status and a truncated, scrubbed body (already available as last_chunk_context) in the error message. Redirects and RETRIABLE_HTTP_CODES are already handled earlier in the block, so this only changes the message for non-retriable errors. Specs in spec/services/import/bulk_imports/http_file_download_strategy_spec.rb would need a case for a 404 with application/json.

A docs update for the troubleshooting section is in progress separately.