Add claims to ID token to support Fulcio integration

In order to add GitLab as a IdP to Fulcio, we need to add some additional claims to our ID token. The following claims should be added:

Field	Description
ci_config_ref_uri	Fully qualified reference to the pipeline definition, defaults to .gitlab-ci.yml. May point to an arbitrary HTTP remote depending on project configuration.
ci_config_sha	Git commit SHA for the pipeline_ref (only populated when pipeline_ref is local to the GitLab instance).
runner_id	ID of the runner.
runner_environment	The type of runner used by the job. May be one of gitlab-hosted or self-hosted.
sha	The commit revision the project is built for.

See #388517 (comment 1342881271) for more context.

• Consolidate id token claims documentation - !117322 (merged)
• Add runner_id, runner_environment, and sha - !116867 (merged)
• Add ci_config_ref_uri for repository config source - !122373 (merged)
• Add ci_config_sha for repository config source - !124647 (merged)

added to epic &10254 (closed)

added WorkingGroupSSCS workflowrefinement labels

assigned to @ali-gitlab

@ali-gitlab, Please add a group or category label to identify issue ownership.

You can refer to the Features by Group handbook page for guidance.

If you are unsure about the correct group, please do not leave the issue without a group label, and refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issue.

This message was generated automatically. You're welcome to improve it.

This issue was automatically tagged with the label groupgeo by TanukiStan, a machine learning classification model, with a probability of 0.55.

If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.

If you are unsure about the correct group, please do not leave the issue without a group label. Please refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issues.

Authors who do not have permission to update labels can leave the issue to be triaged by group leaders initially assigned by TanukiStan

This message was generated automatically. You're welcome to improve it.

added automation:ml groupgeo labels

added devopssystems sectioncore platform labels

changed title from Update CI_JOB_JWT_V2 with new claims to Create CI_JOB_JWT_V3 with new claims

changed the description

changed title from Create CI_JOB_JWT_V3 with new claims to Create CI_JOB_JWT_V3 with additional claims

added workflowin dev label and removed workflowrefinement label

mentioned in merge request !116867 (merged)

changed title from Create CI_JOB_JWT_V3 with additional claims to Add additional claims to CI JWT

mentioned in merge request !117322 (merged)

Since we have the list of ID token claims documented in two different places, I opened Consolidate ID token claims documentation (!117322 - merged) to consolidate the docs.

Consolidate ID token claims documentation (!117322 - merged) is in review.

The first MR to add additional claims is in development: Add runner_id, runner_environment and sha claim... (!116867 - merged). This MR adds runner_id, runner_environment and sha claims to the token.

At least one more MR will be required to add pipeline_ref and pipeline_sha claims.

• Consolidate ID token claims documentation (!117322 - merged) has been merged.
• Add runner_id, runner_environment and sha claim... (!116867 - merged) has been merged.
• Add pipeline_ref and pipeline_sha claims to Ci:... (!117923 - merged) is being implemented.

Overall, this issue is on track to be completed for %16.0.

Verification of Add runner_id, runner_environment and sha claim... (!116867 - merged)

Job https://gitlab.com/ali-gitlab/cosign-demo/-/jobs/4214666702 on a shared runner:

{
  ...
  "runner_id": 12270807,
  "runner_environment": "gitlab-hosted",
  "sha": "31c769dba8987f9dd46c43425f15c8a9ecd4a72a",
  ...
}

runner_id and sha match the job, and runner_environment is correct for a shared runner.

Job https://gitlab.com/ali-gitlab/cosign-demo/-/jobs/4214836495 on a self-hosted runner:

{
  ...
  "runner_id": 23101471,
  "runner_environment": "self-hosted",
  "sha": "9c1a48d7d8e73c6bc2163c62e31329cd0be04bee",
  ...
}

runner_id and sha match the job, and runner_environment is correct for a self-hosted runner.

The final MR for this issue, Add pipeline_ref and pipeline_sha claims to Ci:... (!117923 - merged) has been merged. There is a follow-up issue for making some improvements: Support keyless signing when CI config is locat... (#411317 - closed).

Add pipeline_ref and pipeline_sha claims to Ci:... (!117923 - merged) was reverted due to a high volume of errors in Sentry: 2023-05-19: ActionController::UrlGenerationError (gitlab-com/gl-infra/production#14474 - closed).

MR to reintroduce pipeline_ref and pipeline_sha with FF: Add pipeline_config_url and pipeline_config_sha... (!121597 - closed).

Since our planned implementation has changed a bit, I've closed !121597 (closed) and extracted a smaller MR from it Add `ci_config_ref_uri` claim to Ci::JwtV2 for ... (!122373 - merged). This MR only populates pipeline_ref if the CI config is located in the same repository. In a follow-up MR we can populate it for other config sources as well.

Submitted !122373 (merged) for initial review.

Add `ci_config_ref_uri` claim to Ci::JwtV2 for ... (!122373 - merged) has been merged behind a FF and the flag has been enabled globally.

The final MR for this issue, Add ci_config_sha JWT claim (!124647 - merged), is in review.

Add ci_config_sha JWT claim (!124647 - merged) has been merged but hasn't been deployed yet due to a PCL. Verification is pending.

Opened Add project_visibility JWT claim (!125787 - merged) to add project_visibility claim.

Verification of ci_config_ref_uri and ci_config_sha:

{
  ...
  "sha": "3165b207f5229d5d921f3d4633363130bc3799d4",
  "ci_config_ref_uri": "gitlab.com/ali-gitlab/cosign-demo//.gitlab-ci.yml@refs/heads/main",
  "ci_config_sha": "3165b207f5229d5d921f3d4633363130bc3799d4",
  ...
}

Closing as all relevant MRs have been implemented and verified.

added automation:ml wrong grouppipeline security labels and removed groupgeo label

added devopsverify sectionops labels and removed devopssystems sectioncore platform labels

changed the description

Implementation of pipeline_ref:

project.ci_config_path_or_default could be a file in the project, a file in a different project (optionally on a non-default branch), or a URL to a YAML file. I'm not sure what the value is if AutoDevOps is turned on, but I think we would want this to point to the ADO template.

There are also some other pipeline config sources that may need extra logic. See https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/enums/ci/pipeline.rb#L75 for a list of config_sources.

Docs:

• https://docs.gitlab.com/ee/ci/pipelines/settings.html#specify-a-custom-cicd-configuration-file

Relevant MRs:

• !20179 (merged)
• !68603 (merged)