Azure Stack Hub requires older API version, accessing azure backed object storage fails

Summary

GitLab's object storage implementation fails when using Azure Stack Hub as the storage backend due to API version incompatibility. The current implementation uses newer Azure API versions (2023-08-03) not supported by Azure Stack Hub.

This follows on from a previous request for help (internal link) and is discussed in more detail here (internal link)

The customer would like to move away from Minio, so I wanted to see if we could update things here to resolve this behaviour.

Environment

• GitLab deployment: Kubernetes (AKS)
• Storage backend: Azure Stack Hub Blob Storage
• GitLab version: 17.11.3-ee
• AzCopy version: 10.28.0
• Azure SDK version: azsdk-go-azblob/v1.4.0

Reproduction Steps

1. Configure GitLab to use Azure Blob Storage as object store
2. Attempt to upload artifacts or packages through the web UI
3. Observe 400 error response

Error Details

2025/06/26 12:21:37 ERR: [P#0-T#0] UPLOADFAILED: /srv/gitlab/tmp/backup_tars/1750940407_2025_06_26_17.11.3-ee_gitlab_backup.tar : 400 : 400 The value for one of the HTTP headers is not in the correct format.. When Staging block. X-Ms-Request-Id: 09c8beda-1662-0c33-5b6e-e278e414db5b

Impact

1. Unable to use Azure Stack Hub for object storage
2. Runner logs do not persist when jobs finish
3. Artifacts cannot be uploaded to GitLab
4. Backups fail to capture artifacts stored in alternative storage solutions

Root Cause

GitLab's fork of the Fog module (gitlab-fog-azure-rm) and AzCopy tool use newer Azure API versions (2023-08-03) that are incompatible with Azure Stack Hub, which only supports older API versions (confirmed working with 2020-10-02).

Workarounds Attempted

1. Using MinIO as alternative storage (currently implemented but undesirable due to obsolete version)
2. Setting AZCOPY_DEFAULT_SERVICE_API_VERSION=2020-10-02 (helps with backups but not object storage)
3. Explored PVC options (not viable due to CSI limitations with ReadWriteX access mode)