Release pipeline fails silently when the Tag Creator Token expires: detect it, and announce it in Slack

Summary

The weekly release pipeline has been silently failing to create tags since the Tag Creator Token expired on 2026-08-28. Nothing anywhere announced it. Three consecutive version_tag jobs failed with the same expired-token error, and the first person to notice found out through an unrelated symptom: rake gitlab:indexer:install in GDK aborting with couldn't find remote ref v5.14.12, because the tag the release MR promised had never been created.

Terri raised this in #g_global_search on 2026-09-09 and suggested the tag creator pipeline should also send a note in Slack. This issue proposes that, plus the two adjacent problems the incident exposed.

Evidence

Three version_tag job failures, all identical:

Job	Date	Trace
16233553532	2026-09-01	glab: HTTP 401 … {"error":"invalid_token","error_description":"Token is expired..."}
16335620361	2026-09-07	same
16374580538	2026-09-08	same

Project access token state (GET /projects/2953390/access_tokens): Tag Creator Token id 16056505, access_level: 40, expires_at: 2026-08-28, last_used_at: 2026-08-27T13:14Z — used the day before expiry, then dead. A replacement (id 27413868, expires_at: 2027-04-30) was created on 2026-09-09 and the tag was placed manually, so v5.14.12 now exists. The next expiry is 2027-04-30, and today nothing will tell us when it arrives.

There is no Slack integration configured on this project — GET /projects/2953390/integrations returns []. Same for gitlab-org/gitlab-zoekt-indexer.

Problem 1: a 401 is misread as "the tag does not exist"

.gitlab-ci.yml, version_tag job:

if glab api /projects/$CI_PROJECT_ID/repository/tags/v${VERSION} --silent; then
  echo "Tag v${VERSION} already exists, skipping creation"
else
  echo "Tag v${VERSION} doesn't exist, will create it"

glab api exits non-zero on any failure, so an expired token takes the else branch. The trace shows it plainly:

glab: HTTP 401
Tag v5.14.12 doesn't exist, will create it

The job did not know the tag existed, and it did not know it was unauthenticated. It logged a false conclusion and then failed on the create. The auth failure has to be distinguished from a 404 — check glab auth status (or the tag-read HTTP status) explicitly and fail with a message naming the token, rather than letting an auth error masquerade as a missing tag.

Problem 2: nothing announced the failure

The built-in project-access-token expiry notice emails direct Maintainers and Owners at 60, 30 and 7 days (docs). It is email-only, it goes to individuals rather than the team, and in this case it did not produce action.

Two changes, in increasing cost:

2a. Turn on the GitLab for Slack app for failed pipelines on the default branch. Project → Settings → Integrations → GitLab for Slack app, Failed pipeline event, notify only for the default branch, target #g_global_search. No CI code, no new secret, no security review — it is a first-party integration. This is the change that would actually have caught this incident, and it catches every other release failure too, not just token expiry.

2b. A scheduled token-expiry check that posts to Slack. This is what Terri asked for specifically, and it needs an incoming webhook, which needs a CorpSec application_integration_request — there is no GitLab-maintained Slack notification CI/CD component to adopt (confirmed in the options analysis at https://gitlab.com/gitlab-org/gitlab/-/work_items/612011 : "Checked common-ci-tasks@v4.37, runwayctl/kubernetes-v2-service@v4.62.0, fairway/helm-chart@v1.38.2, and Woodhouse. None of them has a Slack or chat notification component"). The mechanics are a curl of a masked, protected CI variable; notify-slack.sh in island-gitops is the hardened reference implementation (webhook passed via curl --config so it never reaches argv or logs).

The detection query is a GET /projects/:id/access_tokens filtered on expires_at within a threshold — cheap, and it should cover every token this project's CI depends on, not just the tag creator: TAG_CREATOR_TOKEN, INDEXER_RELEASE_TOKEN, and the Danger token.

Problem 3: the same exposure exists in gitlab-zoekt-indexer

gitlab-org/gitlab-zoekt-indexer runs the same weekly-release + TAG_CREATOR_TOKEN shape, has no Slack integration, and its own Tag Creator Token already expired once (id 14766101, expires_at: 2026-06-11). Whatever lands here should land there too.

Proposal

1. Distinguish auth failure from a missing tag in version_tag, and fail with a message that names the token. No approvals needed.
2. Enable the GitLab for Slack app on this project for failed default-branch pipelines → #g_global_search.
3. File the CorpSec incoming-webhook request, then add a scheduled pipeline that reports project access tokens expiring within 45 days to #g_global_search.
4. Apply 1–3 to gitlab-org/gitlab-zoekt-indexer.

Done when

A version_tag job that cannot authenticate says so instead of claiming the tag does not exist; a failed release pipeline on main produces a message in #g_global_search without anyone watching for it; and the 2027-04-30 expiry of Tag Creator Token id 27413868 is announced in that channel before it happens.

Not verified

Whether an existing group-level Slack webhook variable is inherited here — GET /groups/9970/variables returns 403 for a project Maintainer, so group-level CI variables on gitlab-org could not be read. If one already exists, step 3 needs no new CorpSec request. Group and project access token webhooks fire a seven-day expiry event and would remove the polling entirely, but seven days' notice is thinner than the 45-day window proposed above.