[GitLab Migration] Exported relations fail to download if stored in S3 object storage

Whenever GitLab Migration runs and tried to download exported relations from source & source has exported relations stored in AWS S3 - the importer fails to download with the 403 error.

This is happening because in BulkImports::FileDownloadService we first perform a HEAD request to get information about the file, run validations. And AWS S3 does not support HEAD requests to the pre-signed URLs it generates for the exported relations to be accessed, only GET.

curl --head --header "PRIVATE-TOKEN: TOKEN" "http://gdk.test:3000/api/v4/groups/144/export_relations/download?relation=labels" -L --output -
HTTP/1.1 303 See Other

Location: http://127.0.0.1:9000/uploads/bulk_imports/export_upload/export_file/2962/labels.ndjson.gz?X-Amz-Expires=600&X-Amz-Date=20221117T101025Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20221117%2Fgdk%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=fbe75f423f9bed0f84411bef3b82a823633cc892cdc89e27da072e62e778b542

HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 0

Potential fixes

  1. Skip these validations if file is stored in S3
  2. Can we use GET with stream_body: true to get hold of headers and skip downloading body to simulate HEAD request?
  3. Use GET but this makes us download the file twice, so let's not do that 😄
Edited by George Koltsov