Improve process for merging DB migrations

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

The Discord import job has been failing since !2432 (merged) ("Switch discord channels to exclusion list") was merged. The job fails with:

ActiveModel::UnknownAttributeError: unknown attribute 'channel_name' for DiscordMessage.

This is because !2432 (merged) included a database migration that adds the channel_name column to the discord_messages table, but the migration has not yet been run against the database. Database migrations only run when the application is deployed, so any scheduled job that depends on the new schema will fail in the interim.

Failing job: https://gitlab.com/gitlab-org/developer-relations/contributor-success/contributors-gitlab-com/-/jobs/15347860215

Immediate action needed

The pending migration needs to be run / a deployment triggered to unblock the Discord import job.

Problem to solve

We need a better process to prevent this class of failure in the future. When a merge request includes a database migration, there is currently no guardrail or reminder to ensure:

  1. The migration is run (i.e. a deployment is triggered) before dependent scheduled jobs execute.
  2. Contributors are aware of the risk when merging migration-containing MRs.

Implementation plan

Following the discussion, we've agreed on the following phased approach. The root cause is that scheduled pipelines run against main, which can be ahead of the deployed schema. Pointing schedules at a rolling latest tag guarantees they run against already-deployed code.

Phase 1 — Maintain a latest tag (this issue)

Force-update a latest git tag to point at the same commit whenever another (versioned) tag is created. This can be folded into the existing release job (which already runs on $CI_COMMIT_TAG) rather than adding a separate job.

  • Runs as part of the existing release job, on $CI_COMMIT_TAG.
  • Force-updates a lightweight latest tag and pushes it.
  • Requires a token with write_repository scope (CI_JOB_TOKEN alone cannot push).
  • Must not re-trigger itself — use -o ci.skip in the push command (thanks @Taucher2003) so pushing latest does not start a new pipeline.

Phase 2 — Point scheduled pipelines at latest (maintainer follow-up)

Once the latest tag exists, a maintainer updates each schedule in CI/CD → Schedules to run against the latest ref instead of main. This is a one-time manual change per schedule and requires maintainer access.

Phase 3 — Reminder/notification on migration merge (stretch / follow-up)

Add a job on the merge-to-main pipeline that detects a new migration file (under contributors/db/migrate/) and, if found, posts a comment on the merged MR @-mentioning the merger to remind them to consider cutting a tag/release (thanks @mmichaux-ext).

  • Fires after merge, so it blocks nothing.
  • Targets the person who merged.
  • Not failing at merge (a migration MR could never merge) and not auto-deploying (keeps the review gate on the riskiest change type).

Original proposals (for reference)

The following approaches were explored during triage — Phases 1–3 above combine the strongest elements:

1. Notification / reminder on merge (low risk, low effort)

Add a CI check or MR approval rule that detects the presence of a new migration file and posts a reminder comment prompting the author/reviewer to trigger a deployment immediately after merging.

2. Automatic deployment trigger on migration merge (medium risk)

Automatically trigger a tag/release pipeline when a migration is merged to main. Risk: a bad migration could break the live web app without a manual review gate.

3. Decouple scheduled jobs from schema changes (medium effort)

Make scheduled jobs more resilient by checking for expected columns before running, skipping/alerting rather than crashing. The latest tag approach (Phases 1–2) is a stronger variant of this.

4. Migration-aware pipeline gate (medium effort)

Add a CI job that detects pending migrations and fails the pipeline (or pauses the scheduled job) until a deployment has been confirmed.

Next steps

  • Trigger a deployment to run the pending migration and unblock the Discord import job immediately.
  • Phase 1: Add a latest tag force-update step to the existing release job (-o ci.skip to avoid self-triggering).
  • Phase 2 (maintainer): Update all scheduled pipelines to use the latest ref instead of main.
  • Phase 3 (stretch): Add a merge-to-main job that reminds the merger to cut a tag/release when a DB migration is merged.

🤖 This content was generated by GitLab Duo.

Edited by 🤖 GitLab Bot 🤖