Health Check Failing After Upgrade: Database Version Mismatch Due to Post-Migration Handling

Health Check Failing After Upgrade: Database Version Mismatch Due to Post-Migration Handling

Problem Description

After upgrading from GitLab 18.6.2 to 18.7.0, the health check endpoint is failing with the following error:

health_check failed: Current database version (20251214205217) does not match latest migration (20251211100451).

Root Cause Analysis

The issue stems from a mismatch between how the third-party health_check gem and GitLab handle database migrations:

  1. Current database version: 20251214205217 (latest post-deployment migration)
  2. Expected version by health check: 20251211100451 (latest regular migration)

Migration Timeline:

up     20251210130532  regular  18.7         AddKnowledgeGraphEnabledNamespacesNamespaceFk
up     20251211100451  regular  18.7         RemoveZoektMinimumProjectsForTraversalIdSearchFromApplicationSettings   ← Expected by health_check
up     20251027100536  post     18.7         RemoveAgeMeanAgeSumOfSquaresRiskScoreFromVulnerabilityStatistics
up     20251027124625  post     18.7         RemoveRiskScoreFromVulnerabilityNamespaceStatistics
...
up     20251214205217  post     18.7         QueueUpdateDuoSastFpDetectionEnabledToFalse                             ← Actual current version

Technical Details

The third-party health_check gem (version 3.1.0) located at embedded/lib/ruby/gems/3.2.0/gems/health_check-3.1.0/lib/health_check/utils.rb only checks the regular migrations folder (/opt/gitlab/embedded/service/gitlab-rails/db/migrate) and does not account for post-deployment migrations in /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate.

Impact

  • Health check endpoint https://gitlab.example.com/health_check/migrations is non-functional
  • Monitoring systems relying on this endpoint may report false negatives
  • Documentation references an endpoint that doesn't work properly

Proposed Solutions

We need to determine the appropriate course of action:

1. Fix the endpoint

Modify the health check to use a GitLab custom migration checking function that properly handles both regular and post-deployment migrations

The health_check gem allows you to register custom checks. Create an initializer to override the migrations check https://www.rubydoc.info/gems/health_check/3.1.0#label-Checks. 1. Modify the all_migrations check to use a custom class:

# config/initializers/health_check.rb
config.add_custom_check('all_migrations') do   
  Gitlab::HealthCheck::Migrations.check 
end
  1. Have the new custom class check if the current version matches the latest regardless of post deploy source:
# lib/gitlab/health_check/migrations.rb
# Generated partially by DAP - looks right but needs verification
module Gitlab
  module HealthCheck
    class Migrations
      def self.check
        new.check
      end

      def check
        pending = pending_migrations
       
        return '' if pending.empty?
        
        return "Pending migrations:\n  #{pending.map(&:filename).join("\n  ")}"
      end

      private

      def pending_migrations
        regular_pending = pending_in_path('db/migrate')
        post_deploy_pending = pending_in_path('db/post_migrate')
        
        regular_pending + post_deploy_pending
      end

      def pending_in_path(path)
        migrations = ActiveRecord::Migrator.migrations(path)
        current = ActiveRecord::Migrator.current_version
        
        migrations.reject { |m| current >= m.version }
      end
    end
  end
end

Customer Reference

Description was generated using AI

Edited by 🤖 GitLab Bot 🤖