Investigate: Dependency Proxy not working with Azure object storage: "missing signature key" error when pulling

Investigation

In %15.4 we are going to do an analysis of this bug so that we can identify and schedule a fix to this important issue.

Summary

One of our customers (internal) reported that they started to getting an error when pulling an image using the dependency proxy:

Pulling docker image <snipped>/dependency_proxy/containers/gitlab/gitlab-runner-helper:x86_64-e95f89a0 ...
ERROR: Preparation failed: Error response from daemon: missing signature key (docker.go:142:1s)

According to them, they started to experience this issue after they purge the cache in specific group using the API.

I was able to reproduce this on my end (GitLab 14.7.1) after I configured my test instance to use Azure Blob storage and then pull a new image.

Steps to reproduce

  1. Configure Azure Blob storage for Dependency Proxy.
  • Use the consolidated storage options
  1. Pull a new image using the dependency proxy.

What is the current bug behavior?

There is an error when pulling a new docker image:

docker pull 192.168.100.5/ptest/dependency_proxy/containers/gitlab/gitlab-runner-helper:x86_64-e95f89a0
Error response from daemon: missing signature key

What is the expected correct behavior?

The image should be pulled properly.

Relevant logs and/or screenshots

The logs on the GitLab side were not that helpful.

These are the logs from the Docker daemon:

time="2022-02-10T16:18:25.798751692+08:00" level=debug msg="Calling POST /v1.41/images/create?fromImage=192.168.100.5%2Fptest%2Fdependency_proxy%2Fcontainers%2Fgitlab%2Fgitlab-runner-helper&tag=x86_64-e95f89a0"
time="2022-02-10T16:18:25.803146737+08:00" level=debug msg="Trying to pull 192.168.100.5/ptest/dependency_proxy/containers/gitlab/gitlab-runner-helper from https://192.168.100.5 v2"
time="2022-02-10T16:18:25.803744338+08:00" level=warning msg="Error getting v2 registry: Get \"https://192.168.100.5/v2/\": EOF"
time="2022-02-10T16:18:25.803775828+08:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://192.168.100.5/v2/\": EOF"
time="2022-02-10T16:18:25.803799901+08:00" level=debug msg="Trying to pull 192.168.100.5/ptest/dependency_proxy/containers/gitlab/gitlab-runner-helper from http://192.168.100.5 v2"
time="2022-02-10T16:18:34.639108560+08:00" level=debug msg="Fetching manifest from remote" digest="sha256:88226f8898e3e0bae6770d8946d22427c1f2a7859b724206ff53a5d5d69c7c72" error="<nil>" remote="192.168.100.5/ptest/dependency_proxy/containers/gitlab/gitlab-runner-helper:x86_64-e95f89a0"
time="2022-02-10T16:18:37.230210910+08:00" level=info msg="Attempting next endpoint for pull after error: missing signature key"

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Workaround

Use Azure MinIO Gateway instead.

Edited by Tim Rizzi