Gitlab Runner Cache not working with Ceph S3 using signature V2

We are on a self-managed installation of GitLab and Ceph and we are having issues using a distributed cache for our runners, based on S3 buckets on Ceph.

We believe that the problem is due to the fact that (althought is possible to configure the signature V2 from the GitLab Master) at this moment there is no option to configure the signature V2 from the gitlab-runner.

This issue could be related to #27681 and #31034

Our configuration and tests

We are using:

  • GitLab Master 16.2.2
  • GitLab Runner 16.2.0
  • Ceph 17.2

We succesfully configured the Gitlab Master for the same S3 instance and the same bucket (although if the job fails, we have correctly found an execution log at the path /83/f8/83f8[remaining_part_of_the_hash]/[date]/[job_ID]/198401/job.log inside the bucket) but the runner continues to return the error

Creating cache [pipeline_ID]-5-protected...
.nuget/packages: found 4144 matching artifact files and directories
Uploading cache.zip to https://[host]/[bucket-name]/cache/project/[project_id]/[pipeline_ID]-5-protected
FATAL: received: 403 Forbidden
Failed to create cache

This is the configuration (only relevant parts) we used on the GitLab Master inside the gitlab.rb file

gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = true
gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'region' => 'us-east-1',
  'endpoint' => '[ENDPOINT]',
  'aws_access_key_id' => '[ACCESS_KEY_ID]',
  'aws_secret_access_key' => '[SECRET_ACCESS_KEY]',
  'path_style' => true,
  'aws_signature_version' => 2,
  'enable_signature_v4_streaming' => false
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '[BUCKET_NAME]'
gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false
gitlab_rails['object_store']['objects']['lfs']['enabled'] = false
gitlab_rails['object_store']['objects']['uploads']['enabled'] = false
gitlab_rails['object_store']['objects']['packages']['enabled'] = false
gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false
gitlab_rails['object_store']['objects']['pages']['enabled'] = false

and for the GitLab Runner inside the config.toml file

[[runners]]
  executor = "docker"
  [runners.cache]
    MaxUploadedArchiveSize = 0
    Type = "s3"
    Path = "cache"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "[SERVER_ADDRESS]"
      AccessKey = "[ACCESS_KEY]"
      SecretKey = "[SECRET_KEY]"
      BucketName = "[BUCKET_NAME]"
      Insecure = false
  [runners.docker]
    tls_verify = false
    image = "docker:20.10.16"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

We tracked the calls from the machine that hosts our Ceph S3 in order to verify that they are both correctly sent by the runner and received by Ceph, and we also verified that the code returned by Ceph S3 is a 403 - Forbidden.

We tried to perform some operations (like listing buckets, listing, uploading and downloading files) against our S3 Ceph instance with other clients like S3cmd and S3 Browser which both allows to use both version (V2 and V4) for the signature and we verified that:

  • when we set the parameter to use Signature V2 from the clients, all the operations concluded succesfully;
  • when instead, we tried to use the signature V4 from the clients, the response code we received was 403 - Forbidden (which is the same response we are receiveng from the GitLab runner).

Our request

We would like to know if we are missing something in the configuration or if it will be possible to implement a behaviour (like an optional parameter) to allow the GitLab Runner to communicate with S3 compatible instances (like Ceph Object Gateway S3) using the signature V2.

Thanks in advance