## Summary The weekly release pipeline has been silently failing to create tags since the `Tag Creator Token` expired on **2026-08-28**. Nothing anywhere announced it. Three consecutive `version_tag` jobs failed with the same expired-token error, and the first person to notice found out through an unrelated symptom: `rake gitlab:indexer:install` in GDK aborting with `couldn't find remote ref v5.14.12`, because the tag the release MR promised had never been created. Terri raised this in `#g_global_search` on 2026-09-09 and suggested the tag creator pipeline should also send a note in Slack. This issue proposes that, plus the two adjacent problems the incident exposed. ## Evidence Three `version_tag` job failures, all identical: | Job | Date | Trace | |---|---|---| | [16233553532](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/jobs/16233553532) | 2026-09-01 | `glab: HTTP 401` … `{"error":"invalid_token","error_description":"Token is expired..."}` | | [16335620361](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/jobs/16335620361) | 2026-09-07 | same | | [16374580538](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/jobs/16374580538) | 2026-09-08 | same | Project access token state (`GET /projects/2953390/access_tokens`): `Tag Creator Token` id `16056505`, `access_level: 40`, `expires_at: 2026-08-28`, `last_used_at: 2026-08-27T13:14Z` — used the day before expiry, then dead. A replacement (id `27413868`, `expires_at: 2027-04-30`) was created on 2026-09-09 and the tag was placed manually, so `v5.14.12` now exists. The next expiry is **2027-04-30**, and today nothing will tell us when it arrives. There is no Slack integration configured on this project — `GET /projects/2953390/integrations` returns `[]`. Same for `gitlab-org/gitlab-zoekt-indexer`. ## Problem 1: a 401 is misread as "the tag does not exist" `.gitlab-ci.yml`, `version_tag` job: ```yaml if glab api /projects/$CI_PROJECT_ID/repository/tags/v${VERSION} --silent; then echo "Tag v${VERSION} already exists, skipping creation" else echo "Tag v${VERSION} doesn't exist, will create it" ``` `glab api` exits non-zero on **any** failure, so an expired token takes the `else` branch. The trace shows it plainly: ``` glab: HTTP 401 Tag v5.14.12 doesn't exist, will create it ``` The job did not know the tag existed, and it did not know it was unauthenticated. It logged a false conclusion and then failed on the create. The auth failure has to be distinguished from a 404 — check `glab auth status` (or the tag-read HTTP status) explicitly and fail with a message naming the token, rather than letting an auth error masquerade as a missing tag. ## Problem 2: nothing announced the failure The built-in project-access-token expiry notice emails direct Maintainers and Owners at 60, 30 and 7 days ([docs](https://docs.gitlab.com/user/project/settings/project_access_tokens/)). It is email-only, it goes to individuals rather than the team, and in this case it did not produce action. Two changes, in increasing cost: **2a. Turn on the GitLab for Slack app for failed pipelines on the default branch.** Project → Settings → Integrations → [GitLab for Slack app](https://docs.gitlab.com/user/project/integrations/gitlab_slack_application/), `Failed pipeline` event, notify only for the default branch, target `#g_global_search`. No CI code, no new secret, no security review — it is a first-party integration. This is the change that would actually have caught this incident, and it catches every other release failure too, not just token expiry. **2b. A scheduled token-expiry check that posts to Slack.** This is what Terri asked for specifically, and it needs an incoming webhook, which needs a CorpSec [`application_integration_request`](https://gitlab.com/gitlab-com/gl-security/corp/issue-tracker/-/work_items/new?description_template=application_integration_request) — there is no GitLab-maintained Slack notification CI/CD component to adopt (confirmed in the options analysis at https://gitlab.com/gitlab-org/gitlab/-/work_items/612011 : "Checked `common-ci-tasks@v4.37`, `runwayctl/kubernetes-v2-service@v4.62.0`, `fairway/helm-chart@v1.38.2`, and Woodhouse. None of them has a Slack or chat notification component"). The mechanics are a `curl` of a masked, protected CI variable; [`notify-slack.sh` in island-gitops](https://gitlab.com/gitlab-com/gl-security/corp/engineering/island-gitops/-/blob/main/scripts/notify-slack.sh) is the hardened reference implementation (webhook passed via `curl --config` so it never reaches argv or logs). The detection query is a `GET /projects/:id/access_tokens` filtered on `expires_at` within a threshold — cheap, and it should cover every token this project's CI depends on, not just the tag creator: `TAG_CREATOR_TOKEN`, `INDEXER_RELEASE_TOKEN`, and the Danger token. ## Problem 3: the same exposure exists in `gitlab-zoekt-indexer` `gitlab-org/gitlab-zoekt-indexer` runs the same weekly-release + `TAG_CREATOR_TOKEN` shape, has no Slack integration, and its own `Tag Creator Token` already expired once (id `14766101`, `expires_at: 2026-06-11`). Whatever lands here should land there too. ## Proposal 1. Distinguish auth failure from a missing tag in `version_tag`, and fail with a message that names the token. No approvals needed. 2. Enable the GitLab for Slack app on this project for failed default-branch pipelines → `#g_global_search`. 3. File the CorpSec incoming-webhook request, then add a scheduled pipeline that reports project access tokens expiring within 45 days to `#g_global_search`. 4. Apply 1–3 to `gitlab-org/gitlab-zoekt-indexer`. ## Done when A `version_tag` job that cannot authenticate says so instead of claiming the tag does not exist; a failed release pipeline on `main` produces a message in `#g_global_search` without anyone watching for it; and the 2027-04-30 expiry of `Tag Creator Token` id `27413868` is announced in that channel before it happens. ## Not verified Whether an existing group-level Slack webhook variable is inherited here — `GET /groups/9970/variables` returns 403 for a project Maintainer, so group-level CI variables on `gitlab-org` could not be read. If one already exists, step 3 needs no new CorpSec request. Group and project [access token webhooks](https://about.gitlab.com/blog/the-ultimate-guide-to-token-management-at-gitlab/) fire a seven-day expiry event and would remove the polling entirely, but seven days' notice is thinner than the 45-day window proposed above.