Support direct API access to secrets from non-CI/CD workloads

Release notes

GitLab Secrets Manager can now be read by non-CI/CD workloads. A new API endpoint mints a short-lived, project- or group-scoped token that clients present directly to the OpenBao backend to fetch secrets. This unblocks Kubernetes runtime workloads (for example via the External Secrets Operator), Terraform / OpenTofu data sources, and service-account automation, using standard Vault-compatible tooling and a GitLab token the client already has. Available in beta on Premium and Ultimate.

Problem to solve

GitLab Secrets Manager currently supports secret access only through CI/CD pipelines (via Runner JWT) and the GitLab UI (via Rails GraphQL proxy). There is no supported path for non-CI/CD workloads, such as short-lived Kubernetes applications not bound to GitLab, to retrieve secrets programmatically.

As a platform engineer, I want my non-CI workloads to read GitLab-managed secrets using tooling they already have, so I do not have to bind them to a GitLab pipeline or copy secrets into another store.

"Would be nice to be able to use the API to retrieve secrets for non CI/CD processes. We have short lived applications in Kubernetes not bound to GitLab that require access to the vault." — Beta customer

Intended users

User experience goal

A non-CI client can fetch a GitLab-managed secret with tools it already has (curl, a Vault client library, the External Secrets Operator), using a GitLab token it already understands, without GitLab-specific SDKs.

Proposal

A new Rails endpoint mints a short-lived JWT for the caller's project or group and returns the OpenBao connection details (URL, namespace, KV mount, auth mount + role). The client uses the JWT to read directly from OpenBao, the same direct-access pattern Runners already use.

This unblocks the read use cases, which are the actual gap today. Writes already work through existing GraphQL mutations for any user-bound token (PAT, SA token, project/group access token), so they are out of scope here.

This design was reviewed and agreed in the discussion thread below (Fabien, Mark, Joe, Adil).

Endpoint

POST /api/v4/projects/:id/secrets_manager/access_token
POST /api/v4/groups/:id/secrets_manager/access_token

Response shape (aligned with external-secrets.io/v1.VaultProvider, with expires_at at the top level so clients can manage refresh without parsing the JWT):

{
  "expires_at": "2026-05-27T10:35:00Z",
  "provider": {
    "vault": {
      "server": "https://openbao.example.com",
      "namespace": "org_5/ns_42/project_99",
      "path": "secrets/kv",
      "version": "v2",
      "auth": {
        "jwt": {
          "path": "api_jwt",
          "role": "all_api_access",
          "token": "<encoded JWT>"
        }
      }
    }
  }
}

The client can use the JWT two ways:

  • Inline auth: one HTTP call per read. Simplest.
  • Explicit JWT login: one login, many reads. Best for high-fanout like ESO.

Clients use existing Vault client libraries (Vault SDKs, vault CLI, ESO's HashiCorp Vault provider). We are just the JWT issuer. The rest is standard Vault.

Further details

OpenBao auth. We follow the existing one-mount-per-auth-type pattern (pipeline_jwt for runners, user_jwt for the UI) and add a new api_jwt mount with its own CEL role (all_api_access) for non-CI API access. Existing mounts are untouched.

How clients use this endpoint:

  • ESO (K8s runtime). A refresher CronJob calls the endpoint with a stored service-account token and writes the JWT to a K8s Secret. ESO's HashiCorp Vault provider reads that Secret and does explicit JWT login against OpenBao on each reconciliation. Standard ESO config, no custom provider.
  • Terraform. Reads the endpoint at plan and apply time, then uses standard Vault client libraries to log in to OpenBao and read the secret.
  • Technical account ops and admin scripts. A shell script with curl and jq fetches the JWT, then uses inline auth or the vault CLI to read.
Sample: ESO setup (untested, to validate in 19.2) 
# 1. Token Secret (customer-created)
#    Recommended: use a token on a dedicated GitLab service account, not a personal PAT.
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-sm-token
type: Opaque
stringData:
  token: <your-service-account-token>
---
# 2. RBAC for the refresher CronJob
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-sm-jwt-refresher
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitlab-sm-jwt-refresher
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitlab-sm-jwt-refresher
subjects:
  - kind: ServiceAccount
    name: gitlab-sm-jwt-refresher
roleRef:
  kind: Role
  name: gitlab-sm-jwt-refresher
  apiGroup: rbac.authorization.k8s.io
---
# 3. The refresher CronJob: mints a JWT every 3 minutes and writes it to a K8s Secret
apiVersion: batch/v1
kind: CronJob
metadata:
  name: gitlab-sm-jwt-refresher
spec:
  schedule: "*/3 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: gitlab-sm-jwt-refresher
          restartPolicy: OnFailure
          containers:
            - name: refresh
              # Any image with curl, jq, and kubectl works.
              image: alpine/k8s:1.30.0
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -eu
                  JWT=$(curl -sf -X POST \
                    -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
                    "${GITLAB_URL}/api/v4/projects/${PROJECT_ID}/secrets_manager/access_token" \
                    | jq -r .provider.vault.auth.jwt.token)
                  kubectl create secret generic gitlab-sm-jwt \
                    --from-literal=jwt="${JWT}" \
                    --dry-run=client -o yaml | kubectl apply -f -
              env:
                - name: GITLAB_TOKEN
                  valueFrom: { secretKeyRef: { name: gitlab-sm-token, key: token } }
                - name: GITLAB_URL
                  value: https://gitlab.example.com
                - name: PROJECT_ID
                  value: "12345"
---
# 4. ESO SecretStore: points at OpenBao and reads the refreshed JWT
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gitlab-openbao
spec:
  provider:
    vault:
      server: https://openbao.example.com
      namespace: org_5/ns_42/project_99
      path: secrets/kv
      version: v2
      auth:
        jwt:
          path: api_jwt
          role: all_api_access
          secretRef:
            name: gitlab-sm-jwt
            key: jwt

Customers then write standard ExternalSecret resources pointing at gitlab-openbao with data/explicit/<name> as the remote key. ESO references: HashiCorp Vault provider, SecretStore API, ExternalSecret API.

Sample: Terraform workaround (untested, until &21177 lands) 
variable "gitlab_token" { sensitive = true }
variable "gitlab_url"   { default = "https://gitlab.example.com" }
variable "project_id"   {}

# Step 1: Mint a JWT via curl wrapped in an external data source.
data "external" "gitlab_sm_jwt" {
  program = ["sh", "-c", <<-EOT
    curl -sf -X POST \
      -H "PRIVATE-TOKEN: ${var.gitlab_token}" \
      "${var.gitlab_url}/api/v4/projects/${var.project_id}/secrets_manager/access_token" \
      | jq '{token: .provider.vault.auth.jwt.token}'
  EOT
  ]
}

# Step 2: Configure the standard Vault provider with the JWT.
provider "vault" {
  address   = "https://openbao.example.com"
  namespace = "org_5/ns_42/project_99"

  auth_login_jwt {
    path = "api_jwt"
    role = "all_api_access"
    jwt  = data.external.gitlab_sm_jwt.result.token
  }
}

# Step 3: Read the secret with the stock Vault data source.
data "vault_kv_secret_v2" "db_password" {
  mount = "secrets/kv"
  name  = "explicit/DB_PASSWORD"
}

resource "aws_db_instance" "example" {
  password = data.vault_kv_secret_v2.db_password.data["value"]
  # ...
}

State file caveat. Terraform writes every value it reads to its state file. Once a resource references the secret, the value lives in state. This is a general Terraform property, not specific to our endpoint. Customers who cannot accept the state-file copy should not reference SM secrets in long-lived Terraform resources.

Permissions and Security

Two layers of authorization:

  1. Minting a token reuses the existing :read_project_secrets ability (and group equivalent). The role lookup goes through max_member_access_for_*, which already respects service account scoping (TLG, subgroup, project).
  2. Reading a value is governed by a new, explicit read_value secret permission (see below). Membership alone does not expose values. An Owner must deliberately grant read_value to a principal. This is enforced in OpenBao: only the value-read policy grants read on the data path.

The minted JWT carries an auth_via claim (PAT / PrAT / GrAT) for audit forensics.

New read_value permission. Today the per-principal secret permissions are read (metadata only), write, and delete. None grant read on the value path, so values are not readable outside the CI flow. We add an explicit value-read permission and, to remove the "read what?" ambiguity, relabel the existing one:

Permission Action OpenBao capability Path
Read metadata (today's "Read") read_metadata read metadata path
Write write create + update value path
Delete delete delete value path
Read value (new) read_value read value path

read_value is additive and requires read_metadata (you cannot read a value for a secret you cannot otherwise see). No data migration: actions are derived from OpenBao capabilities on readback, so a read capability on the metadata path maps to read_metadata and one on the value path maps to read_value. Existing grants keep working and gain nothing until someone opts in.

Policy structure (two policies per principal). To keep the value read off the UI mount, each principal gets two OpenBao policies:

  • A management policy (under users/) used by the existing user mount: metadata read, write, delete. It never grants read on the value path, so the UI flow still cannot read values.
  • A dedicated read-only policy (under api/) used by the new API mount: read on the value path only, written when read_value is granted and removed when it is not.

Each mount's CEL attaches its own policy. The user mount attaches the management policy, the API mount attaches the api/ read-only policy. So an API token can read values but never write, regardless of the principal's other grants. This api/ policy is also the surface we would extend if we later support API writes (the same mount, with write capabilities added).

Expected impact by membership level:

  • No access (0): cannot mint a token. Endpoint returns 404.
  • Guest (10): cannot mint a token.
  • Reporter (20) and up: can mint a token. Can read values only for secrets where they (or a role/group/member-role they belong to) were granted read_value.

Security notes:

  • The minted JWT is short-lived (proposed 5 min TTL) and scoped by CEL to the issuing project or group. A stolen JWT only grants the value reads the principal was explicitly granted, and only for that one project or group.
  • Value access is opt-in per principal via read_value. Membership does not implicitly expose values, so this is secure by default.
  • Auth accepts any user-bound token (PAT, SA token, project/group access token). No new auth mechanism. The realistic attack surface (a stolen long-lived token) is unchanged from today's GraphQL path.
  • A dedicated token scope (read_secrets_via_openbao) is deferred to GA, coordinated with groupauthorization.
  • A threat model will be done with the Application Security Team (@gitlab-com/gl-security/appsec) as part of the implementation.

Documentation

  • New API docs page for the endpoint (request, response shape, beta notice).
  • A "non-CI access" guide with the ESO refresher recipe and the Terraform workaround.
  • Both ship in the same milestone as the endpoint.

Availability & Testing

No QA / E2E work is in scope for the endpoint. The existing SM RSpec suite boots a real OpenBao server via the :gitlab_secrets_manager tag, so the full mint-then-read path is covered at the request-spec level.

  • Unit: new JWT class (claims, auth_via, TTL), CEL role program.
  • Integration (request specs, :gitlab_secrets_manager):
    • New endpoint contract: response shape, authz (:read_project_secrets reuse), feature flag off, beta, expired token, unenrolled namespace.
    • JWT-against-OpenBao validation: extend ee/spec/requests/secrets_management/authentication_boundaries_spec.rb. It already builds each JWT type, logs into the booted OpenBao with use_cel_auth: true, and asserts the CEL role accepts or rejects and grants the right policies. We add the new api_jwt mount + all_api_access role as new rows, plus the negative cases (wrong project, wrong scope).
    • Provisioning: service spec asserting the api_jwt mount + role are created on SM provision.
  • End-to-end (follow-up): a QA spec covering the real deployed read path (call the endpoint via the API, then read from the real external OpenBao URL with the JWT). This mirrors project_secret_ci_access_success_spec.rb but for the API path instead of the pipeline path. Tracked as a separate issue. The full ESO-in-a-cluster path is not automated (the QA framework has no ESO harness); the ESO flow is validated manually with design partners during 19.2 beta.

Availability risk is low. The endpoint is behind a feature flag, off by default, and the new OpenBao mount is additive (existing pipeline_jwt and user_jwt mounts are untouched).

Available Tier

Premium and Ultimate.

Feature Usage Metrics

  • Internal event on token issuance (count of mint calls per project / group).
  • Read volume via the direct-OpenBao path, from audit logs (tracked in #600967).

What does success look like, and how can we measure that?

Success metrics:

  • Adoption: number of projects / groups minting access tokens.
  • At least one design-partner customer running ESO against the endpoint in 19.2.

Acceptance criteria:

  • An authorized caller (Reporter+) gets a valid JWT plus OpenBao connection details.
  • A caller granted read_value can read those secret values through the api_jwt mount. A caller without read_value cannot read values, even with project membership.
  • An unauthorized caller (Guest or no access) gets 404. A caller on a non-enrolled namespace gets 404. With the feature flag off, the endpoint is not exposed.
  • The JWT expires at the advertised expires_at.

What is the type of buyer?

Platform / infrastructure and security buyers (the same buyers who adopt Secrets Manager). This is an enablement feature for SM, gated to Premium and Ultimate.

Is this a cross-stage feature?

Yes. It touches groupauthorization for token scopes and fine-grained PATs on service accounts, and the fulfillment work (usage tracking and billing) tracked in #600967.

What is the competitive advantage or differentiation for this feature?

Direct, Vault-compatible secret access with no proprietary SDK or agent lock-in. Customers reuse the HashiCorp Vault ecosystem they already know (ESO provider, Vault SDKs, vault CLI) and a GitLab token they already have. Reads are decoupled from Rails, so secret availability scales with OpenBao independently of GitLab.

Scope and breakdown

This issue covers the endpoint plus provisioning for new SM enrollments, so a freshly provisioned project or group works end to end.

Implementation plan (this issue):

  1. Configurable TTL on GlobalSecretsManagerJwt (today hardcoded to 30s).
  2. New read_value permission in the model (relabel existing read to read_metadata), plus the policy wiring so read_value grants read on the value path.
  3. New JWT class emitting the api scope and auth_via claim.
  4. New api_jwt OpenBao mount + CEL role in the provision service (new enrollments get it). The role surfaces the per-principal value-read policies.
  5. New endpoint (project + group) reusing :read_project_secrets for token issuance, behind the feature flag, marked beta.
  6. Request specs (endpoint contract + authentication_boundaries_spec.rb extension) and provisioning service spec.
  7. Docs + ESO refresher recipe.

Backend only. The read_value GraphQL exposure and the UI to grant it land in #602726 (frontend), which does not block this issue from merging. Until that ships, read_value is grantable in specs and via the API, just not through the UI.

Split out as sibling issues:

  • #602726 — "Read value" permission in the secrets permission UI (frontend + GraphQL exposure).
  • #602549 — backfill the api_jwt mount + role into already-enrolled SM namespaces. Operational catch-up for existing customers, separate rollout risk.
  • #602550 — QA spec for the non-CI API read path (endpoint -> real external OpenBao read).

Maturity

Endpoint ships as beta (route_setting :lifecycle, :beta), aligned with SM's Open Beta status. It stays beta for one milestone past SM GA to firm up the spec from design-partner feedback.

Consumers (separate work items)

This endpoint is the shared primitive. Each consumer ships on its own track.

  • Kubernetes Operator — &20382
  • GitLab Terraform Provider — &21177
  • glab CLI — to be opened in gitlab-org/cli. Not required for Phase 1; curl-based YAML covers the urgent ESO path.
  • Workload identity federation (OIDC inbound) — #601894. Exchanges an external OIDC token (K8s SA, AWS/GCP/Azure) for an SM JWT, removing the stored long-lived credential. Future phase.

Links / references

Edited by 🤖 GitLab Bot 🤖