2021-07-15: Authentication Errors Pulling Container Registry Images

added IncidentActive Source::IMAIncidentDeclare incident severity2 labels

assigned to @nnelson and @cdu1

changed the severity to High - S2

Slack channel here.

Production checks fail because there are blockers

Production has active incidents

Production has no change requests in progress

GitLab Deployment Health Status - overview

cny-api
cny-git
cny-web
main-api
main-git
main-sidekiq
main-web

no active deployment

Link to failing job indicating docker login failure: https://gitlab.com/gitlab-qa-sandbox-group/qa-test-2021-07-15-20-30-18-fe986954512344f0/npm-project-d2d0657cbc921b3f/-/jobs/1428673591

This project was scheduled for deletion, but failed with the following message: PG::SyntaxError: ERROR: each UNION query must have the same number of columns LINE 3: ..."namespaces"."id" IN (4909902, 12742051))) SELECT "members".... ^

The job could have likely failed because it was on a new project which automatically inherited the project-setting job_token_scope_enabled: true

[ gprd ] production> Project.find(28178930).ci_cd_settings.job_token_scope_enabled
=> true

Running with gitlab-runner 14.0.1 (c1edb478)
  on qa-runner-1626381200 _4iQRFx4
Resolving secrets 00:00
Preparing the "docker" executor 00:00
Using Docker executor with image node:14-buster ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478 ...
WARNING: Failed to pull image with policy "always": Error response from daemon: pull access denied for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:205:0s)
ERROR: Job failed (system failure): failed to pull image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-c1edb478" with specified policies [always]: Error response from daemon: pull access denied for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper, repository does not exist or may require 'docker login': denied: requested access to the resource is denied (manager.go:205:0s)

Source: job

added NeedsRootCause label

added NeedsService label

added NeedsCorrectiveActions label

changed the description

{
  "_index": "pubsub-registry-inf-gprd-001069",
  "_type": "_doc",
  "_id": "fL8OrHoBuGSvJmprhHVH",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-07-15T21:24:29.721Z",
    "kubernetes": {
      "region": "us-east1-d",
      "namespace_name": "gitlab",
      "container_image": "dev.gitlab.org:5005/gitlab/charts/components/images/gitlab-container-registry:v3.5.2-gitlab",
      "pod_name": "gitlab-registry-5b8b87df49-jk6g9",
      "host": "gke-gprd-us-east1-d-registry-1-ef830985-18ht",
      "container_name": "registry"
    },
    "publish_time": "2021-07-15T21:24:29.605Z",
    "json": {
      "migrating_repository": false,
      "use_database": false,
      "write_fs_metadata": true,
      "type": "registry",
      "msg": "repository name not known to registry",
      "time": "2021-07-15T21:24:24Z",
      "tag": "gitlab_registry.var.log.containers.gitlab-registry-5b8b87df49-jk6g9_gitlab_registry-606cfa119dac41916b934cfff38d8011389e6220e899755b2a8cebd94371309a.log",
      "root_repo": "gitlab-qa-sandbox-group",
      "vars_name": "gitlab-qa-sandbox-group/qa-test-2021-07-15-20-30-18-fe986954512344f0/npm-project-d2d0657cbc921b3f",
      "shard": "default",
      "environment": "gprd",
      "correlation_id": "01FAP0WVNFY3AE6Q2WEKVBEAGS",
      "tier": "sv",
      "go_version": "go1.16.4",
      "stage": "main",
      "level": "error",
      "code": "NAME_UNKNOWN",
      "detail": "map[name:gitlab-qa-sandbox-group/qa-test-2021-07-15-20-30-18-fe986954512344f0/npm-project-d2d0657cbc921b3f]",
      "auth_user_name": "",
      "error": "name unknown: repository name not known to registry"
    },
    "type": "pubsubbeat-pubsub-registry-inf-gprd-75bfff974f-nw65j",
    "host": {
      "name": "pubsubbeat-pubsub-registry-inf-gprd-75bfff974f-nw65j"
    }
  },
  "fields": {
    "json.time": [
      "2021-07-15T21:24:24.000Z"
    ]
  },
  "highlight": {
    "json.detail": [
      "map[name:gitlab-qa-sandbox-group/qa-test-2021-07-15-20-30-18-fe986954512344f0/@kibana-highlighted-field@npm@/kibana-highlighted-field@-@kibana-highlighted-field@project@/kibana-highlighted-field@-@kibana-highlighted-field@d2d0657cbc921b3f@/kibana-highlighted-field@]"
    ],
    "json.vars_name": [
      "gitlab-qa-sandbox-group/qa-test-2021-07-15-20-30-18-fe986954512344f0/@kibana-highlighted-field@npm@/kibana-highlighted-field@-@kibana-highlighted-field@project@/kibana-highlighted-field@-@kibana-highlighted-field@d2d0657cbc921b3f@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1626384264000
  ]
}

mentioned in issue on-call-handovers#1847 (closed)

One customer observed the same failure gitlab-org/gitlab#336287 (comment 628121380)

changed the description

mentioned in issue gitlab-org/gitlab#336289 (closed)

mentioned in issue gitlab-org/quality/pipeline-triage#91 (closed)

Diff from canary and production https://gitlab.com/gitlab-org/security/gitlab/compare/0cc4a51c8c9...b21159d5b23

Still debugging and it appears to be a container registry authentication issue.

changed the description

changed title from 2021-07-15: QA failing on canary to 2021-07-15: Authentication Errors Pulling Container Registry Images

This is failing reliably:

https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/jobs/1429098747

mentioned in issue gitlab-org/gitlab#336287 (closed)

The problem likely is due to gitlab-org/gitlab!65848 (merged).

I can reproduce this reliably via the Rails console on a running build:

app.get('https://gitlab.com/jwt/auth?scope=repository:gitlab-org/cluster-integration/auto-build-image:pull&service=container_registry&account=gitlab-ci-token', headers: { 'Authorization' => 'Basic REDACTED' })

...where REDACTED is the base64-encoded value of gitlab-ci-token:SOME TOKEN (though I had some trouble using base64 on the command-line, so I just used curl -v using http://username:password@host).

The build_read_container_image permission check (https://gitlab.com/gitlab-org/gitlab/blob/29cf01b56a73bd59924cd9119de73e4bf0e2d4f1/app/services/auth/container_registry_authentication_service.rb#L193) is failing due to project_allowed_for_job_token check:

- [0] prevent when all?(anonymous, ~public_project) ((@anton : Project/10400718))
+ [0] prevent when ~project_allowed_for_job_token ((@anton : Project/10400718))
  [0] enable when can?(:public_user_access) ((@anton : Project/10400718))
  [0] enable when all?(guest, can?(:read_container_image)) ((@anton : Project/10400718))

In the working case:

irb(main):211:0> policy.debug(:build_read_container_image)
- [0] prevent when all?(anonymous, ~public_project) ((@anton : Project/10400718))
- [0] prevent when ~project_allowed_for_job_token ((@anton : Project/10400718))
+ [0] enable when can?(:public_user_access) ((@anton : Project/10400718))
=> #<DeclarativePolicy::Runner::State:0x00007f5992ca4bd8 @enabled=true, @prevented=false>

It appears that this feature is designed to restrict CI access to that project, but it is also restricting access to public projects.

Other way to reproduce the issue:

runner_project = Project.find_by_full_path('gitlab-org/cluster-integration/auto-build-image')
project = Project.find_by_full_path('anton/incident-5174-test')
build = Ci::Build.find(1428929992) # Mark this running if it's not
user = User.find_by_username('anton')

RequestStore.begin! # This is needed for the CI scope token
auth_result = Gitlab::Auth.find_for_git_client('gitlab-ci-token', build.token, project: nil, ip: '127.0.0.1')
auth_params = { scopes: ['repository:gitlab-org/cluster-integration/auto-build-image:pull'], service: 'container_registry' } 
result = ::Auth::ContainerRegistryAuthenticationService.new(auth_result.project, auth_result.actor, auth_params).execute(authentication_abilities: auth_result.authentication_abilities)
result[:token] # This is the JWT that can be decoded in https://jwt.io

Disabling the ci_scoped_job_token feature flag appears to have resolved the issue for our first test project.

gitlab-org/gitlab#332272 (closed) is the rollout issue

added IncidentMitigated label and removed IncidentActive label

We've disabled the relevant feature flag back which has resolved these errors.

mentioned in issue gitlab-org/gitlab#332272 (closed)

changed the description

added IncidentResolved label and removed IncidentMitigated label

@fabiopitino It seems that ci_scoped_job_token is related to this incident, so maybe we should also revert gitlab-org/gitlab!65848 (merged) before 14.1 gets cut so that it doesn't get shipped in the next self-managed release.

/cc @gitlab-org/release/managers

@tkuah I'm preparing a revert of gitlab-org/gitlab!65848 (merged) which enabled the FF. I'll investigate what happened.

/cc @gitlab-org/release/managers

Revert MR gitlab-org/gitlab!66305 (merged)

This incident was automatically closed because it has the IncidentResolved label.

Note: All incidents are closed automatically when they are resolved, even when there is a pending review. Please see the Incident Workflow section on the Incident Management handbook page for more information.

closed

mentioned in issue on-call-handovers#1848 (closed)

mentioned in merge request gitlab-org/gitlab!66305 (merged)

The new permission model for CI job token scope should have not impacted existing projects unless the project setting was enabled. We had enabled the feature flag the day before but we didn't observe issues with it be cause all projects had the setting disabled by default.

With gitlab-org/gitlab!65848 (merged) we enabled the setting by default for new projects and it seems when we started seeing issues. The QA failures on Canary could be caused by the fact that new projects had enabled job token scope and by default don't have access to all projects unless allowlisted in the job token scope (via project settings).

I'm wondering whether we observed this issue with existing projects, which may suggest that some logic got leaked outside the FF and project setting guards.

The project setting controls which projects are allowlisted if accessed via CI_JOB_TOKEN.
The feature flag controls whether the project setting is visible to users and whether authentications occur via CI_JOB_TOKEN - This is still the ultimate control point.

By disabling the feature flag we stopped tracking whether current_user was being authenticated via CI_JOB_TOKEN. This prevents any other permissions restrictions that could have been applied by the job token scope.

Right, I think we saw this with new projects trying to fetch container registry images hosted by existing projects. #5174 (comment 628261169) has more details.

Things to still investigate:

were existing projects impacted? This suggests yes: https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/jobs/1429098747
were resources with public access impacted? This was by design as per PoC.

@jreporter @cheryl.li with regards to public access resources we would need to revisit this part of the permissions. Similar discussion gitlab-org/gitlab#332272 (comment 628098918). I think we would need to make a change where public access resources are excluded from job token scope enforcement.

Will investigate the rest on Monday.

I think we would need to make a change where public access resources are excluded from job token scope enforcement.

That makes sense, should I create a separate issue for tracking @fabiopitino ?

@jreporter Yes please Thank you.

Created gitlab-org/gitlab#336398 (closed)

@jreporter @cheryl.li as per #5174 (comment 630173506) we haven't found (yet) data confirming that existing projects were impacted. This means that the rollout worked OK but there were some issues.

So far the issues observed during the rollout were 2:

Public projects should not be counted against the job token scope. Although this was by design we realized that it causes more harm than good. Remediation: gitlab-org/gitlab#336398 (closed)
QA failures were legit due to the fact that we create new projects all the time and those got the job token scope feature enabled by default - Will open an issue to fix the QA tests when we rollout the feature again.

Okay got it, thanks for the confirmation. @fabiopitino

Does this mean we can resume rollout once we implement the fix in 1?

@jreporter I think so.

mentioned in issue gitlab-org/gitlab#334432 (closed)

mentioned in issue gitlab-org/gitlab#336398 (closed)

@sabrams @10io could you help me understand how to trace authentication for docker pull after a successful docker login authentication using CI_JOB_TOKEN? I'm trying to figure out why this job was failing.

Do we use a different authentication route than other API requests?

@fabiopitino Here are the interaction flows: https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/auth-request-flow.md

The authentication is implemented by the https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/jwt_controller.rb.

The tag being accessed by the job doesn't exist in the registry: https://log.gprd.gitlab.net/goto/4f5e5527aeb85bf41f869a6069999dbc (internal). The registry is replying 404 all the times.

I'm not sure if that helps but aren't gitlab-assets-ee tags usually commit shas? I'm a bit surprised to see what seems to be a branch name.

@fabiopitino On the bottom of my note in #5174 (comment 628261169) is the sequence I used to reproduce the failure from the command-line. This simulates this authentication flow:

GitLab CI build starts running.
Runner attempts to pull an image from Docker with no JWT. The registry returns a 401 Unauthenticated.
Runner authenticates with /jwt/auth via HTTP Basic Authentication with gitlab-ci-token as the username and build token as the password. It passes a scope in the form repository:<project name>:pull (e.g. repository:gitlab-org/cluster-integration/auto-build-image:pull).
JwtController#auth returns a JWT with authorized actions. Note that even if no actions are allowed, the controller will return a 200.
In the working case, the JwtController returns a valid JWT with actions set (https://gitlab.com/gitlab-org/gitlab/blob/29cf01b56a73bd59924cd9119de73e4bf0e2d4f1/app/services/auth/container_registry_authentication_service.rb#L121) to repository:gitlab-org/cluster-integration/auto-build-image:pull if allowed.
In the non-working case, the actions is blank because can_access? in https://gitlab.com/gitlab-org/gitlab/blob/29cf01b56a73bd59924cd9119de73e4bf0e2d4f1/app/services/auth/container_registry_authentication_service.rb#L107 returns false.

That's why the problem boils down to that project_allowed_for_job_token policy check. That seems to take precedence over can?(:public_user_access)

@stanhu Yes, that makes sense. In #5174 (comment 628261169) project_allowed_for_job_token returned false because the project anton/incident-5174-test was a newly created project that inherited the project CI/CD setting job_token_scope_enabled: true. This means that by default a CI_JOB_TOKEN from anton/incident-5174-test can only access resources from the same project unless other projects are added/allowlisted to the job token scope.

As you highlighted, the problem with this is that public_user_access is not taken in consideration. We are fixing this.

I'm trying to verify whether existing projects where impacted, which should have had the job_token_scope_enabled: false by default. For example this job failure was linked in this issue but I'm also seeing the same job succeeding around the same time https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/jobs/1429136807 - cc @tkuah

@fabiopitino It is not confirmed that https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/jobs/1429098747 is related to this incident. As someone pointed out it has different error messages. Sorry about the noise.

@tkuah @fabiopitino I've encountered https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/jobs/1429098747 before. It's not related to this issue. Note that the manifest is missing, not unauthorized. It looks like someone is trying to build a omnibus-gitlab package with a custom branch, but this doesn't work because the fetch-assets job attempts to retrieve a image with the branch name. In the past, I've hacked this by modifying fetch-assets to look at master (https://dev.gitlab.org/gitlab/omnibus-gitlab/-/merge_requests/200/diffs).