Skip to content

E2E pipelines - 403 Forbidden - No access granted when fabricating project runners

Across a number of E2E tests in production we are intermittently seeing the following when a test fabricates a project runner:

Resource at https://gitlab.com/api/v4/runners/23017652?private_token=[MASKED] could not be found (403): `{"message":"403 Forbidden - No access granted"}`.

For example https://ops.gitlab.net/gitlab-org/quality/production/-/jobs/9906880

Failure/Error:
            Resource::ProjectRunner.fabricate! do |runner|
              runner.project = project
              runner.name = runner_name
              runner.tags = [runner_name]
              runner.executor = :docker
            end
          
          QA::Resource::Errors::ResourceNotFoundError:
            Resource at https://gitlab.com/api/v4/runners/23017652?private_token=[MASKED] could not be found (403): `{"message":"403 Forbidden - No access granted"}`.

These are logged in Kibana but don't provide a lot of info eg.

https://log.gprd.gitlab.net/app/discover#/?_a=%28index:%277092c4e2-4eb5-46f2-8305-a7da2edad090%27%2Cquery%3A%28language%3Akuery%2Cquery%3A%27json.correlation_id%20%3A%20c96bb1ed7ca804ed7edcc889e3603380%27%29%29&_g=%28time%3A%28from%3A%272023-04-27T12:00:36.056Z%27%2Cto%3A%272023-04-28T12:00:36.056Z%27%29%29

Related failures

Substantial troubleshooting done in #403174 (closed)

#409072 (closed)

#408953 (closed)

#408948 (closed)

#408947 (closed)

#408806 (closed)

#408781 (closed)

#408779 (closed)

#408734 (closed)

#408714 (closed)

#408713 (closed)

#408710 (closed)

#408709 (closed)

#408645 (closed)

#408638 (closed)

#408513 (closed)

#408512 (closed)

#408511 (closed)

#408510 (closed)

#408509 (closed)

#408498 (closed)

#408393 (closed)

#408331 (closed)

#408330 (closed)

#408319 (closed)

#408293 (closed)

#408291 (closed)

#408290 (closed)

#408287 (closed)

#408271 (closed)

#408270 (closed)

#408269 (closed)

#408240 (closed)

#408050 (closed)

#408049 (closed)

#408048 (closed)

#408047 (closed)

among others...

Edited by Mark Lapierre