Docker added an [API for creating Org Access Tokens (OAT)](https://docs.docker.com/reference/api/hub/latest/#tag/org-access-tokens) in the past year. Use this (ideally via a Terraform provider) to rotate the OAT automatically. _This ticket was created from_ [_INC-9793_](https://app.incident.io/gitlab/incidents/9793) _using_ [_incident.io_](https://app.incident.io) 🔥 --- ## Why During incident INC-9793 (2026-05-04), the Docker Organization Access Token (OAT) used by GitLab's Dependency Proxy silently expired. This token — stored as "Docker gitlabci OAT" in 1Password and set as `GITLAB_DEPENDENCY_TOKEN` in the `gitlab-org` and `gitlab-com` groups — had no automated rotation, no expiry alerting, and no monitoring. The result was a Severity 2 incident that blocked CI pipelines across all internal engineering teams. ## What Implement **automated rotation** of the Docker OAT using the Docker Hub API, ideally managed via Terraform. The rotated token value must be automatically synced to the `GITLAB_DEPENDENCY_TOKEN` CI/CD variable in both the `gitlab-org` and `gitlab-com` groups. ## How ### Approach The Docker Hub API supports creating and revoking OATs: https://docs.docker.com/reference/api/hub/latest/#tag/org-access-tokens There are two viable implementation paths — choose whichever fits the existing infrastructure patterns: **Option A: Terraform (preferred if a Docker Hub provider exists)** - Check if the [Docker Hub Terraform provider](https://registry.terraform.io/providers/BarnabyShearer/dockerhub/latest) or a community equivalent supports OAT management. - If yes, define the OAT as a Terraform resource and use a scheduled pipeline or `atlantis` to rotate it on a schedule (e.g., every 90 days). - Store the output token in a secret manager (e.g., GCP Secret Manager or 1Password via the 1Password Terraform provider) and sync it to the GitLab CI/CD variable. **Option B: Scheduled CI pipeline script** - Create a scheduled pipeline in an appropriate ops repository (e.g., `gitlab-com/gl-infra/ops` or a dedicated credentials rotation repo). - The pipeline script should: 1. Call the Docker Hub API to create a new OAT: `POST https://hub.docker.com/v2/orgs/{org}/tokens` 2. Update the `GITLAB_DEPENDENCY_TOKEN` variable in `gitlab-org` via the GitLab API: `PUT /api/v4/groups/gitlab-org/variables/GITLAB_DEPENDENCY_TOKEN` 3. Update the same variable in `gitlab-com`: `PUT /api/v4/groups/gitlab-com/variables/GITLAB_DEPENDENCY_TOKEN` 4. Revoke the old OAT via the Docker Hub API: `DELETE https://hub.docker.com/v2/orgs/{org}/tokens/{uuid}` 5. Update the value in 1Password (optional, but recommended for auditability). - Schedule this pipeline to run every 60–80 days (Docker OATs can be set to expire at 90 days). ### Implementation steps 1. **Investigate** whether a Terraform provider for Docker Hub OAT management exists and is usable in the current infra setup. Check `https://registry.terraform.io/providers/BarnabyShearer/dockerhub/latest/docs`. 2. **Identify the correct repository** for this automation. Look at how other credential rotation jobs are structured in `gitlab-com/gl-infra` (e.g., search for existing scheduled pipelines that rotate tokens). 3. **Implement** the chosen approach (Terraform or scheduled pipeline). 4. **Add monitoring:** Set an alert or expiry check so that if the token is within 14 days of expiry and rotation has not occurred, an alert fires to `#production` or creates an incident. 5. **Document** the automation in the runbook at `docs/package-registry/dependency-proxy-for-containers-runbook.md` in [gitlab-com/runbooks](https://gitlab.com/gitlab-com/runbooks) — specifically update the `## Credential Rotation → Automation` section added by [production-engineering#28869](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28869). ### Key API references - Create OAT: `POST https://hub.docker.com/v2/orgs/gitlab/tokens` with body `{"label": "gitlabci-oat", "scopes": ["repo:read"], "expires_in": 7776000}` (90 days in seconds) - List OATs: `GET https://hub.docker.com/v2/orgs/gitlab/tokens` - Delete OAT: `DELETE https://hub.docker.com/v2/orgs/gitlab/tokens/{uuid}` - GitLab Group Variable API: `PUT /api/v4/groups/:id/variables/:key` ### Acceptance criteria - [ ] The Docker OAT is rotated automatically without manual intervention. - [ ] The new token is synced to `GITLAB_DEPENDENCY_TOKEN` in both `gitlab-org` and `gitlab-com` groups. - [ ] The old token is revoked after successful rotation. - [ ] An alert or check exists to catch rotation failures before the token expires. - [ ] The runbook is updated to reflect that rotation is now automated.