Establish Ownership, Process, and SLOs for DB Migration Squashing

Context

Follow up to issue #81 (comment 2475486069)

Problem Statement

The runtime of RSpec background migration jobs and related jobs (e.g., db:check:schema) is increasing linearly with the growing number of DB migrations in our codebase and affects pipeline runtime across all tiers. This indicates a maintenance problem that requires regular squashing of DB migrations. However, we currently lack a sustainable approach to DB migration squashing that addresses the challenges of context, ownership, process, and scope.

Current Situation

  1. The latest DB migration squashing was performed in gitlab-org/quality/engineering-productivity/team#564 (closed) and some MRs for this squash contained over 1,000 file changes!
  2. Key challenges identified:
    1. Context: DB migrations involve changes specific to multiple stages and groups; no single team can gather all context
    2. Ownership: DevEx teams are not positioned as appropriate owners as they lack necessary context.
    3. Process: Incomplete documentation, tooling, and processes mean migrations must be done manually.
    4. Scope: Squashing requires updating code references and rewriting RSpecs to match new schema, crossing multiple team boundaries.

Impact

This initiative addresses several critical pain points affecting team performance and our core values of results, iteration, and efficiency.

  1. Blocked iteration cycles
    1. Unrelated test failures and long running Rspec migrations delay MR approvals.
    2. Engineers spend valuable time waiting rather than delivering incremental value.
  2. Delayed Incident Response
    1. In case of master broken issues (Ex: 1, 2), teams must either bypass running critical tests or wait longer
    2. This directly impacts our ability to quickly address and resolve incidents
  3. Capacity issues and Context switching: When DB squashing does occur, it forces team members to:
    1. Create and review overly complex, large MRs
    2. Context-switch away from their planned work
    3. Divert capacity from value-adding tasks to maintenance work
  4. System performance degradation: Our current reactive approach results in:
    1. Steadily increasing pipeline runtimes across all tiers
    2. Accumulating technical debt
    3. Growing long-term maintenance burden and performance issues.

Solution

Short term

Manually https://docs.gitlab.com/development/database/migration_squashing/

Long term

We need to establish:

  1. Shared Ownership Model:
    • DevEx teams to build necessary tooling and processes
    • Stage/group teams to handle migrations related to their areas
    • Clear responsibility across cross-stage migrations
  2. Standardized Process:
    • Documented procedure for DB migration squashing
    • Defined SLOs for migration frequency (e.g., monthly, quarterly )
    • Automated tooling to assist with migration squashing (e.g related discussion)
  3. Proactive Approach:
    • Regular scheduled migration squashing instead of reactive approach
    • Monitoring system to track growth of migrations and alert when approaching thresholds

Goals

  • Identify stakeholders from each stage/group to participate in defining ownership model
  • Document current process based on recent migration squashing work
  • Define SLOs for DB migration squashing (frequency, timeline, etc.)
  • Explore automation opportunities to reduce manual effort
  • Create proposal for shared ownership model

Related work

Edited by Abhinaba Ghosh