Draft: Report invalid indexes and hint at fixing them

What does this MR do and why?

Follow-up to !259890 (merged), which made schema validation treat invalid indexes as absent.

  • The missing-index message says how to fix it. While an invalid index holds an expected name, the CREATE INDEX from structure.sql fails. MissingIndexes now adds a hint to its message: REINDEX INDEX CONCURRENTLY when the invalid index has the structure.sql definition, otherwise DROP INDEX CONCURRENTLY and create the index from structure.sql. gitlab:db:validate_schema prints it.
  • Invalid leftovers are reported as extra again. An invalid index that structure.sql lacks, such as a *_ccnew leftover of reindexing, still costs writes and disk space. ExtraIndexes reports it, and so does the index inconsistency metric in Service Ping. The other validators still treat invalid indexes as absent.
  • The index query starts from pg_index. It joins by OID instead of by name through pg_indexes, and selects whether each index is invalid. Database#indexes returns the valid indexes and Database#invalid_indexes the invalid ones. On the GDK test and development databases, the rewritten query returned the same indexes as the old one.

The commits follow these points in reverse order.

References

Screenshots or screen recordings

No UI change. gitlab:db:validate_schema output for an invalid index_users_on_name:

Before After
The index index_users_on_name is missing from the database The index index_users_on_name is missing from the database. An invalid index holds this name. Rebuild it with REINDEX INDEX CONCURRENTLY public.index_users_on_name

How to set up and validate locally

  1. In a Rails console on GDK, mark an index invalid, add an invalid leftover, and run the index validators. The transaction rolls both changes back:

    conn = ApplicationRecord.connection
    validation = Gitlab::Schema::Validation
    
    conn.transaction do
      conn.execute(<<~SQL)
        UPDATE pg_index SET indisvalid = false WHERE indexrelid = 'public.index_users_on_name'::regclass;
        CREATE INDEX index_users_on_name_ccnew ON users (name);
        UPDATE pg_index SET indisvalid = false WHERE indexrelid = 'public.index_users_on_name_ccnew'::regclass;
      SQL
    
      database = validation::Sources::Database.new(conn)
      structure_sql = validation::Sources::StructureSql.new(Rails.root.join('db/structure.sql'))
    
      [validation::Validators::MissingIndexes, validation::Validators::ExtraIndexes].each do |validator|
        puts validator.new(structure_sql, database).execute.map(&:error_message).grep(/index_users_on_name/)
      end
    
      raise ActiveRecord::Rollback
    end
  2. MissingIndexes reports index_users_on_name with the REINDEX hint, and ExtraIndexes reports index_users_on_name_ccnew. On master, there is no hint and index_users_on_name_ccnew isn't reported.

  3. For the drop hint, add DROP INDEX public.index_users_on_name; CREATE INDEX index_users_on_name ON users (username); before the first UPDATE and run it again.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Niko Belokolodov

Merge request reports

Loading
Loading