Job using JWT/OIDC ran fine on trunk, fails on tag pipeline

Hey, I am a 101% puzzled by this behaviour. I don't know whether there's some kind of bug lurking there or if I'm just stupid. Here's what's happening:

We used to deploy trunk (master) straight to prod. The corresponding jobs authenticate against AWS via CI_JOB_JWT_V2 or the new GITLAB_OIDC_TOKEN. We now deploy to prod on tags, nothing else really changed, and it broke. I now see

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

and

Unable to locate credentials. You can configure credentials by running "aws configure".

and in later terraform steps

Error: No valid credential sources found

The exact same jobs run. It's the same environment. I checked the role, it's still the same. The script portion itself is just terraform init plan and apply. When trying GITLAB_OIDC_TOKEN I add

  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

to the job spec and the auth script can handle both env vars. This works just fine in other projects.

There's also two projects with the exact same behaviour. I even created a branch, dropped the rules from the deployment job and it did successfully auth agains AWS and deploy the stuff. Nothing else changed.

I don't see what should be different for a pipeline triggered by a tag than one for a push to any branch or MR.

Any ideas?

Assignee Loading
Time tracking Loading