Merge branch '384592-aw-jwt-docs' into 'master'

Add docs for authenticating with ID tokens

See merge request !108330



Merged-by: Marcel Amirault <mamirault@gitlab.com>
Approved-by: Marcel Amirault <mamirault@gitlab.com>
Reviewed-by: Avielle Wolfe <awolfe@gitlab.com>
Reviewed-by: Marcel Amirault <mamirault@gitlab.com>
Co-authored-by: Avielle Wolfe <awolfe@gitlab.com>

doc/ci/cloud_services/index.md

@@ -8,15 +8,30 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 
 > - `CI_JOB_JWT` variable for reading secrets from Vault [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) in GitLab 12.10.
 > - `CI_JOB_JWT_V2` variable to support additional OIDC providers [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346737) in GitLab 14.7.
+> - [ID tokens](../yaml/index.md) to support any OIDC provider, including HashiCorp Vault, [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356986) in GitLab 15.7.
 
-GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/connect/faq/) that allows your build and deployment job access to cloud credentials and services. Historically, teams stored secrets in projects or applied permissions on the GitLab Runner instance to build and deploy. To support this, a predefined variable named `CI_JOB_JWT_V2` is included in the CI/CD job allowing you to follow a scalable and least-privilege security approach.
+GitLab CI/CD supports [OpenID Connect (OIDC)](https://openid.net/connect/faq/) to
+give your build and deployment jobs access to cloud credentials and services.
+Historically, teams stored secrets in projects or applied permissions on the GitLab Runner
+instance to build and deploy. OIDC capable [ID tokens](../yaml/index.md#id_tokens) are configurable
+in the CI/CD job allowing you to follow a scalable and least-privilege security approach.
+
+In GitLab 15.6 and earlier, you must use `CI_JOB_JWT_V2` instead of an ID token,
+but it is not customizable. In GitLab 14.6 an earlier you must use the `CI_JOB_JWT`, which has limited support.
 
 ## Requirements
 
 - Account on GitLab.
 - Access to a cloud provider that supports OIDC to configure authorization and create roles.
 
-The original implementation of `CI_JOB_JWT` supports [HashiCorp Vault integration](../examples/authenticating-with-hashicorp-vault/index.md). The updated implementation of `CI_JOB_JWT_V2` supports additional cloud providers with OIDC including AWS, Azure, GCP, and Vault.
+ID tokens and `CI_JOB_JWT_V2` support cloud providers with OIDC, including:
+
+- AWS
+- Azure
+- GCP
+- HashiCorp Vault
+
+The `CI_JOB_JWT` only supports the [HashiCorp Vault integration](../examples/authenticating-with-hashicorp-vault/index.md).
 
 NOTE:
 Configuring OIDC enables JWT token access to the target environments for all pipelines.
@@ -25,10 +40,6 @@ review for the pipeline, focusing on the additional access. You can use the [sof
 as a starting point, and for more information about supply chain attacks, see
 [How a DevOps Platform helps protect against supply chain attacks](https://about.gitlab.com/blog/2021/04/28/devops-platform-supply-chain-attacks/).
 
-The `CI_JOB_JWT_V2` variable is available for testing, but the full feature is planned
-to be generally available when [issue 360657](https://gitlab.com/gitlab-org/gitlab/-/issues/360657)
-is complete.
-
 ## Use cases
 
 - Removes the need to store secrets in your GitLab group or project. Temporary credentials can be retrieved from your cloud provider through OIDC.
@@ -39,18 +50,18 @@ is complete.
 
 ## How it works
 
-Each job has a JSON web token (JWT) provided as a CI/CD [predefined variable](../variables/predefined_variables.md) named `CI_JOB_JWT` or `CI_JOB_JWT_V2`. This JWT can be used to authenticate with the OIDC-supported cloud provider such as AWS, Azure, GCP, or Vault.
+Each job can be configured with ID tokens, which are provided as a CI/CD variable. These JWTs can be used to authenticate with the OIDC-supported cloud provider such as AWS, Azure, GCP, or Vault.
 
 The following fields are included in the JWT:
 
 | Field                   | When   | Description |
 | ----------------------- | ------ | ----------- |
+| `aud`                   | Always | Specified in the [ID tokens](../yaml/index.md#id_tokens) configuration |
 | `jti`                   | Always | Unique identifier for this token |
 | `iss`                   | Always | Issuer, the domain of your GitLab instance |
 | `iat`                   | Always | Issued at |
 | `nbf`                   | Always | Not valid before |
 | `exp`                   | Always | Expires at |
-| `aud`                   | Always | Issuer, the domain of your GitLab instance |
 | `sub`                   | Always |`project_path:{group}/{project}:ref_type:{type}:ref:{branch_name}` |
 | `namespace_id`          | Always | Use this to scope to group or user level namespace by ID |
 | `namespace_path`        | Always | Use this to scope to group or user level namespace by path |
@@ -72,7 +83,7 @@ The following fields are included in the JWT:
 {
   "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558",
   "iss": "https://gitlab.example.com",
-  "aud": "https://gitlab.example.com",
+  "aud": "https://vault.example.com",
   "iat": 1585710286,
   "nbf": 1585798372,
   "exp": 1585713886,
@@ -102,8 +113,8 @@ sequenceDiagram
     participant GitLab
     Note right of Cloud: Create OIDC identity provider
     Note right of Cloud: Create role with conditionals
-    Note left of GitLab: CI/CD job with CI_JOB_JWT_V2
-    GitLab->>+Cloud: Call cloud API with CI_JOB_JWT_V2
+    Note left of GitLab: CI/CD job with ID token
+    GitLab->>+Cloud: Call cloud API with ID token
     Note right of Cloud: Decode & verify JWT with public key (https://gitlab/-/jwks)
     Note right of Cloud: Validate audience defined in OIDC
     Note right of Cloud: Validate conditional (sub, aud) role
@@ -115,14 +126,25 @@ sequenceDiagram
 
 1. Create an OIDC identity provider in the cloud (for example, AWS, Azure, GCP, Vault).
 1. Create a conditional role in the cloud service that filters to a group, project, branch, or tag.
-1. The CI/CD job includes a predefined variable `CI_JOB_JWT_V2` that is a JWT token. You can use this token for authorization with your cloud API.
+1. The CI/CD job includes an ID token which is a JWT token. You can use this token for authorization with your cloud API.
 1. The cloud verifies the token, validates the conditional role from the payload, and returns a temporary credential.
 
 ## Configure a conditional role with OIDC claims
 
-To configure the trust between GitLab and OIDC, you must create a conditional role in the cloud provider that checks against the JWT token. The condition is validated against the JWT to create a trust specifically against two claims, the audience and subject.
+To configure the trust between GitLab and OIDC, you must create a conditional role in the cloud provider that checks against the JWT.
+The condition is validated against the JWT to create a trust specifically against two claims, the audience and subject.
+
+- Audience or `aud`: Configured as part of the ID token:
+
+  ```yaml
+  job_needing_oidc_auth:
+    id_tokens:
+      OIDC_TOKEN:
+        aud: https://oidc.provider.com
+    script:
+      - echo $OIDC_TOKEN
+  ```
 
-- Audience or `aud`: The URL of the GitLab instance. This is defined when the identity provider is first configured in your cloud provider.
 - Subject or `sub`: A concatenation of metadata describing the GitLab CI/CD workflow including the group, project, branch, and tag. The `sub` field is in the following format:
   - `project_path:{group}/{project}:ref_type:{type}:ref:{branch_name}`
 

doc/ci/secrets/id_token_authentication.md

+---
+stage: Verify
+group: Pipeline Authoring
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+type: tutorial
+---
+
+# OpenID Connect (OIDC) Authentication Using ID Tokens **(FREE)**
+
+You can authenticate with third party services using GitLab CI/CD's
+[ID tokens](../yaml/index.md#id_tokens).
+
+## ID Tokens
+
+[ID tokens](../yaml/index.md#id_tokens) are JSON Web Tokens (JWTs) that can be added to a GitLab CI/CD job. They can be used for OIDC
+authentication with third-party services, and are used by the [`secrets`](../yaml/index.md#secrets) keyword to authenticate with HashiCorp Vault.
+
+ID tokens are configured in the `.gitlab-ci.yml`. For example:
+
+```yaml
+job_with_id_tokens:
+  id_tokens:
+    FIRST_ID_TOKEN:
+      aud: https://first.service.com
+    SECOND_ID_TOKEN:
+      aud: https://second.service.com
+  script:
+    - first-service-authentication-script.sh $FIRST_ID_TOKEN
+    - second-service-authentication-script.sh $SECOND_ID_TOKEN
+```
+
+In this example, the two tokens have different `aud` claims. Third party services can be configured to reject tokens
+that do not have an `aud` claim matching their bound audience. Use this functionality to reduce the number of
+services with which a token can authenticate. This reduces the severity of having a token compromised.
+
+## Manual ID Token authentication
+
+You can use ID tokens for OIDC authentication with a third party service. For example:
+
+```yaml
+manual_authentication:
+  variables:
+    VAULT_ADDR: http://vault.example.com:8200
+  image: vault:latest
+  id_tokens:
+    VAULT_ID_TOKEN:
+      aud: http://vault.example.com:8200
+  script:
+    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-example jwt=$VAULT_ID_TOKEN)"
+    - export PASSWORD="$(vault kv get -field=password secret/myproject/example/db)"
+    - my-authentication-script.sh $VAULT_TOKEN $PASSWORD
+```
+
+## Automatic ID Token authentication with HashiCorp Vault **(PREMIUM)**
+
+You can use ID tokens to automatically fetch secrets from HashiCorp Vault with the
+[`secrets`](../yaml/index.md#secrets) keyword.
+
+### Enable automatic ID token authentication
+
+To enable automatic ID token authentication:
+
+1. On the top bar, select **Main menu > Projects** and find your project.
+1. On the left sidebar, select **Settings > CI/CD**.
+1. Expand **Token Access**.
+1. Toggle **Limit JSON Web Token (JWT) access** to enabled.
+
+### Configure automatic ID Token authentication
+
+If one ID token is defined, the `secrets` keyword automatically uses it to authenticate with Vault. For example:
+
+```yaml
+job_with_secrets:
+  id_tokens:
+    VAULT_ID_TOKEN:
+      aud: https://example.vault.com
+  secrets:
+    PROD_DB_PASSWORD:
+      vault: example/db/password # authenticates using $VAULT_ID_TOKEN
+  script:
+    - access-prod-db.sh --token $PROD_DB_PASSWORD
+```
+
+If more than one ID token is defined, use the `token` keyword to specify which token should be used. For example:
+
+```yaml
+job_with_secrets:
+  id_tokens:
+    FIRST_ID_TOKEN:
+      aud: https://first.service.com
+    SECOND_ID_TOKEN:
+      aud: https://second.service.com
+  secrets:
+    FIRST_DB_PASSWORD:
+      vault: first/db/password
+      token: $FIRST_ID_TOKEN
+    SECOND_DB_PASSWORD:
+      vault: second/db/password
+      token: $SECOND_ID_TOKEN
+  script:
+    - access-first-db.sh --token $FIRST_DB_PASSWORD
+    - access-second-db.sh --token $SECOND_DB_PASSWORD
+```

doc/ci/secrets/index.md

@@ -23,10 +23,16 @@ GitLab has selected [Vault by HashiCorp](https://www.vaultproject.io) as the
 first supported provider, and [KV-V2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2)
 as the first supported secrets engine.
 
-GitLab authenticates using Vault's
+By default, GitLab authenticates using Vault's
 [JSON Web Token (JWT) authentication method](https://developer.hashicorp.com/vault/docs/auth/jwt#jwt-authentication), using
-the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`)
-introduced in GitLab 12.10.
+the [JSON Web Token](https://gitlab.com/gitlab-org/gitlab/-/issues/207125) (`CI_JOB_JWT`).
+
+[ID tokens](../yaml/index.md#id_tokens) is the preferred secure way to authenticate with Vault,
+because ID tokens are defined per-job. GitLab can also authenticate with Vault by using the `CI_JOB_JWT`,
+but that token is provided to every job, which can be a security risk.
+
+The [Authenticating and Reading Secrets With HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md)
+tutorial has more details about authenticating with ID tokens.
 
 You must [configure your Vault server](#configure-your-vault-server) before you
 can use [use Vault secrets in a CI job](#use-vault-secrets-in-a-ci-job).
@@ -106,9 +112,14 @@ After [configuring your Vault server](#configure-your-vault-server), you can use
 the secrets stored in Vault by defining them with the `vault` keyword:
 
 ```yaml
-secrets:
-  DATABASE_PASSWORD:
-    vault: production/db/password@ops  # translates to secret `ops/data/production/db`, field `password`
+job_using_vault:
+  id_tokens:
+    VAULT_ID_TOKEN:
+      aud: https://gitlab.com
+  secrets:
+    DATABASE_PASSWORD:
+      vault: production/db/password@ops  # translates to secret `ops/data/production/db`, field `password`
+      token: $VAULT_ID_TOKEN
 ```
 
 In this example:
@@ -125,9 +136,13 @@ To overwrite the default behavior, set the `file` option explicitly:
 
 ```yaml
 secrets:
+  id_tokens:
+    VAULT_ID_TOKEN:
+      aud: https://gitlab.com
   DATABASE_PASSWORD:
     vault: production/db/password@ops
     file: false
+    token: $VAULT_ID_TOKEN
 ```
 
 In this example, the secret value is put directly in the `DATABASE_PASSWORD` variable
@@ -177,8 +192,8 @@ Always restrict your roles to a project or namespace by using one of the provide
 claims like `project_id` or `namespace_id`. Without these restrictions, any JWT
 generated by this GitLab instance may be allowed to authenticate using this role.
 
-For a full list of `CI_JOB_JWT` claims, read the
-[How it works](../examples/authenticating-with-hashicorp-vault/index.md#how-it-works) section of the
+For a full list of ID token JWT claims, read the
+[How It Works](../examples/authenticating-with-hashicorp-vault/index.md#how-it-works) section of the
 [Authenticating and Reading Secrets With HashiCorp Vault](../examples/authenticating-with-hashicorp-vault/index.md) tutorial.
 
 You can also specify some attributes for the resulting Vault tokens, such as time-to-live,
@@ -188,7 +203,7 @@ for the JSON web token method.
 
 ## Using a self-signed Vault server
 
-When the Vault server is using a self-signed certificate, you will see the following error in the job logs:
+When the Vault server is using a self-signed certificate, you see the following error in the job logs:
 
 ```plaintext
 ERROR: Job failed (system failure): resolving secrets: initializing Vault service: preparing authenticated client: checking Vault server health: Get https://vault.example.com:8000/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299: x509: certificate signed by unknown authority
@@ -197,7 +212,7 @@ ERROR: Job failed (system failure): resolving secrets: initializing Vault servic
 You have two options to solve this error:
 
 - Add the self-signed certificate to the GitLab Runner server's CA store.
-  If you deployed GitLab Runner using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html), you will have to create your own GitLab Runner image.
+  If you deployed GitLab Runner using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html), you have to create your own GitLab Runner image.
 - Use the `VAULT_CACERT` environment variable to configure GitLab Runner to trust the certificate:
   - If you are using systemd to manage GitLab Runner, see [how to add an environment variable for GitLab Runner](https://docs.gitlab.com/runner/configuration/init.html#setting-custom-environment-variables).
   - If you deployed GitLab Runner using the [Helm chart](https://docs.gitlab.com/runner/install/kubernetes.html):

doc/ci/yaml/index.md

@@ -3794,6 +3794,43 @@ job:
 - The `file` keyword is a setting for the CI/CD variable and must be nested under
   the CI/CD variable name, not in the `vault` section.
 
+#### `secrets:token`
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356986) in GitLab 15.8.
+
+Use `secrets:token` to explicitly select a token to use when authenticating with Vault by referencing the token's CI/CD variable.
+
+This keyword has no effect if [**Limit JSON Web Token (JWT) access**](../secrets/id_token_authentication.md#enable-automatic-id-token-authentication)
+is disabled.
+
+**Keyword type**: Job keyword. You can use it only as part of a job.
+
+**Possible inputs**:
+
+- The name of an ID token
+
+**Example of `secrets:token`**:
+
+```yaml
+job:
+  id_tokens:
+    AWS_TOKEN:
+      aud: https://aws.example.com
+    VAULT_TOKEN:
+      aud: https://vault.example.com
+  secrets:
+    DB_PASSWORD:
+      vault: gitlab/production/db
+      token: $VAULT_TOKEN
+```
+
+**Additional details**:
+
+- When the `token` keyword is not set and **Limit JSON Web Token (JWT) access** enabled, the first ID token
+  is used to authenticate.
+- When **Limit JSON Web Token (JWT) access** is disabled, the `token` keyword is ignored and the `CI_JOB_JWT`
+  CI/CD variable is used to authenticate.
+
 ### `services`
 
 Use `services` to specify any additional Docker images that your scripts require to run successfully. The [`services` image](../services/index.md) is linked