Skip to content

fix: When using credential store, get creds for specified index

Jim Park requested to merge jim80net/gitlab-runner:jim80net/credential_store_gets_specified_index into main

What does this MR do?

This MR updates auth_config.go such that the functions for Credential Stores pass in and get authorization tokens for the indexes as supplied, rather than solely relying on the credential store to list all of its known indexes.

Why was this MR needed?

We use a lightweight wrapper around https://github.com/awslabs/amazon-ecr-credential-helper and found that when using this as a credential store, that unless the cache was already populated with the ECR repository that we wanted access to, the docker executor would only populate credentials retrieved from the list command.

Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Running with gitlab-runner development version (HEAD)  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]:   on ci12 docker 02bcba73                           job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Shell configuration: environment: []
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: dockercommand:
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: - sh
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: - -c
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash \nelif [ -x /usr/bin/bash
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: ]; then\n\texec /usr/bin/bash \nelif [ -x /bin/bash ]; then\n\texec /bin/bash \nelif
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh \nelif [ -x /usr/bin/sh
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: ]; then\n\texec /usr/bin/sh \nelif [ -x /bin/sh ]; then\n\texec /bin/sh \nelif [
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: -x /busybox/sh ]; then\n\texec /busybox/sh \nelse\n\techo shell not found\n\texit
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: 1\nfi\n\n"
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: command: bash
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: arguments: []
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: passfile: false
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: extension: ""
Dec 13 20:06:26 ci12.local gitlab-runner[6141]:   job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Using Docker executor with image 1234567890.dkr.ecr.us-east-2.amazonaws.com/example-default:79bf7000 ...  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Looking for prebuilt image gitlab/gitlab-runner-helper:x86_64-latest...  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Creating user-defined volumes...                    job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Using container "fc14065cb82c27b1db71916d5fba33817dc4a082a1d429b9f485883b0d5eb7d8" as cache "/cache"...  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Using host-based "/etc/vault" for "/etc/vault"...   job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Creating build volume...                            job=2294996 project=776 runner=02bcba73
Dec 13 20:06:26 ci12.local gitlab-runner[6141]: Error while inspecting "runner-02bcba73-project-776-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8" container: Error: No such container: runner-02bcba73-project-776-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8 (cache_container.go:48:0s)  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:27 ci12.local gitlab-runner[6141]: Starting cache container "53753e6093a5ab9851595834811569f7dd24c853cdcc472d4881e727634b7301"...  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:27 ci12.local gitlab-runner[6141]: Waiting for cache container "53753e6093a5ab9851595834811569f7dd24c853cdcc472d4881e727634b7301"...  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:27 ci12.local gitlab-runner[6141]: Waiting for container 53753e6093a5ab9851595834811569f7dd24c853cdcc472d4881e727634b7301 ...  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:27 ci12.local gitlab-runner[6141]: Feeding runners to channel                          builds=1
Dec 13 20:06:28 ci12.local gitlab-runner[6141]: Using container "53753e6093a5ab9851595834811569f7dd24c853cdcc472d4881e727634b7301" as cache "/builds"...  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:28 ci12.local gitlab-runner[6141]: Creating services...                                job=2294996 project=776 runner=02bcba73
Dec 13 20:06:28 ci12.local gitlab-runner[6141]: Starting Docker command...                          job=2294996 project=776 runner=02bcba73
Dec 13 20:06:28 ci12.local gitlab-runner[6141]: Looking for prebuilt image gitlab/gitlab-runner-helper:x86_64-latest...  job=2294996 project=776 runner=02bcba73
time="2019-12-13T20:06:28Z" level=debug msg="Listing credentials"
Dec 13 20:06:29 ci12.local gitlab-runner[6141]: Appending trace to coordinator... ok                code=202 job=2294996 job-log=0-262 job-status=running runner=02bcba73 sent-log=0-261 status=202 Accepted
time="2019-12-13T20:06:29Z" level=debug msg="Retrieving credentials" region=us-east-2 registry=0987654321 serverURL="https://0987654321.dkr.ecr.us-east-2.amazonaws.com"
time="2019-12-13T20:06:29Z" level=debug msg="Checking file cache" registry=0987654321
time="2019-12-13T20:06:29Z" level=debug msg="Using cached token" registry=0987654321
time="2019-12-13T20:06:30Z" level=debug msg="Listing credentials"
time="2019-12-13T20:06:31Z" level=debug msg="Retrieving credentials" region=us-east-2 registry=0987654321 serverURL="https://0987654321.dkr.ecr.us-east-2.amazonaws.com"
time="2019-12-13T20:06:31Z" level=debug msg="Checking file cache" registry=0987654321
time="2019-12-13T20:06:31Z" level=debug msg="Using cached token" registry=0987654321
Dec 13 20:06:31 ci12.local gitlab-runner[6141]: No credentials found for 1234567890.dkr.ecr.us-east-2.amazonaws.com  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:31 ci12.local gitlab-runner[6141]: Removed container 53753e6093a5ab9851595834811569f7dd24c853cdcc472d4881e727634b7301 with <nil>  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:31 ci12.local gitlab-runner[6141]: WARNING: Preparation failed: Error response from daemon: Get https://1234567890.dkr.ecr.us-east-2.amazonaws.com/v2/example-default/manifests/79bf7000: no basic auth credentials (executor_docker.go:190:0s)  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:31 ci12.local gitlab-runner[6141]: WARNING: Preparation failed: Error response from daemon: Get https://1234567890.dkr.ecr.us-east-2.amazonaws.com/v2/example-default/manifests/79bf7000: no basic auth credentials (executor_docker.go:190:0s)  job=2294996 project=776 runner=02bcba73
Dec 13 20:06:31 ci12.local gitlab-runner[6141]: Will be retried in 3s ...                           job=2294996 project=776 runner=02bcba73
Dec 13 20:06:31 ci12.local gitlab-runner[6141]: Will be retried in 3s ...                           job=2294996 project=776 runner=02bcba73

Interspersed in the logs is the logs from the credential helper, showing that it only gets list and get operators for repositories existing in cache, and not the ECR repo that needs the authentication token. The token for 0987654321 is cached, whereas 1234567890 is not. The authentication, and therefore the build, fails as a consequence.

When the cache.json contains a cached copy of the ECR credentials or the docker client is used to perform the pull directly, then the issue does not exhibit.

With the fix, the index is passed to the credential helper, and a get is performed, which is consistent with the design of the credential store.

Dec 14 04:21:01 ci12.local gitlab-runner[32596]: Appending trace to coordinator... ok                code=202 job=2296259 job-log=0-262 job-status=running runner=02bcba73 sent-log=0-261 status=202 Accepted
time="2019-12-14T04:21:01Z" level=debug msg="Listing credentials"
completed action for list : unused
starting action for get : https://0987654321.dkr.ecr.us-east-2.amazonaws.com
time="2019-12-14T04:21:02Z" level=debug msg="Retrieving credentials" region=us-east-2 registry=0987654321 serverURL="https://0987654321.dkr.ecr.us-east-2.amazonaws.com"
time="2019-12-14T04:21:02Z" level=debug msg="Checking file cache" registry=0987654321
time="2019-12-14T04:21:02Z" level=debug msg="Using cached token" registry=0987654321
completed action for get : https://0987654321.dkr.ecr.us-east-2.amazonaws.com
starting action for get : 1234567890.dkr.ecr.us-east-2.amazonaws.com
time="2019-12-14T04:21:03Z" level=debug msg="Retrieving credentials" region=us-east-2 registry=1234567890 serverURL=1234567890.dkr.ecr.us-east-2.amazonaws.com
time="2019-12-14T04:21:03Z" level=debug msg="Checking file cache" registry=1234567890
time="2019-12-14T04:21:03Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=1234567890
time="2019-12-14T04:21:03Z" level=debug msg="Saving credentials to file cache" registry=1234567890
completed action for get : 1234567890.dkr.ecr.us-east-2.amazonaws.com
Dec 14 04:21:03 ci12.local gitlab-runner[32596]: Authenticating with credentials from $DOCKER_AUTH_CONFIG  job=2296259 project=776 runner=02bcba73
Dec 14 04:21:03 ci12.local gitlab-runner[32596]: Using AWS to connect to  in order to resolve 1234567890.dkr.ecr.us-east-2.amazonaws.com/smtprelay-default:79bf7000 ...  job=2296259 project=776 runner=02bcba73
D

Again, output from the credential-helper is interspersed. You can see that the GetAll operation is performed, then a Get immediately follows. This enables the build to continue as designed.

Are there points in the code the reviewer needs to double check?

Yes, it might be quicker and simpler to perform a simple Get rather than a GetAll followed by a Get, but wasn't confident in what unknown impacts that might have on intended behavior. I'm open to feedback on this, and any other, matter.

Does this MR meet the acceptance criteria?

  • Documentation created/updated - N/A. No functional change
  • Added tests for this feature/bug - No but I don't believe auth_config.go is well tested, and if this is a requirement, I could use some help to properly write a test.
  • In case of conflicts with master - branch was rebased - N/A No conflicts.

What are the relevant issue numbers?

#4426

Merge request reports