Draft: Report invalid indexes and hint at fixing them
What does this MR do and why?
Follow-up to !259890 (merged), which made schema validation treat invalid indexes as absent.
- The missing-index message says how to fix it. While an invalid index holds an expected name, the
CREATE INDEXfromstructure.sqlfails.MissingIndexesnow adds a hint to its message:REINDEX INDEX CONCURRENTLYwhen the invalid index has thestructure.sqldefinition, otherwiseDROP INDEX CONCURRENTLYand create the index fromstructure.sql.gitlab:db:validate_schemaprints it. - Invalid leftovers are reported as extra again. An invalid index that
structure.sqllacks, such as a*_ccnewleftover of reindexing, still costs writes and disk space.ExtraIndexesreports it, and so does the index inconsistency metric in Service Ping. The other validators still treat invalid indexes as absent. - The index query starts from
pg_index. It joins by OID instead of by name throughpg_indexes, and selects whether each index is invalid.Database#indexesreturns the valid indexes andDatabase#invalid_indexesthe invalid ones. On the GDK test and development databases, the rewritten query returned the same indexes as the old one.
The commits follow these points in reverse order.
References
- Review threads on !259890 (merged): missing-index hint, extra-index signal, query from
pg_index - Related to #632819
Screenshots or screen recordings
No UI change. gitlab:db:validate_schema output for an invalid index_users_on_name:
| Before | After |
|---|---|
The index index_users_on_name is missing from the database |
The index index_users_on_name is missing from the database. An invalid index holds this name. Rebuild it with REINDEX INDEX CONCURRENTLY public.index_users_on_name |
How to set up and validate locally
-
In a Rails console on GDK, mark an index invalid, add an invalid leftover, and run the index validators. The transaction rolls both changes back:
conn = ApplicationRecord.connection validation = Gitlab::Schema::Validation conn.transaction do conn.execute(<<~SQL) UPDATE pg_index SET indisvalid = false WHERE indexrelid = 'public.index_users_on_name'::regclass; CREATE INDEX index_users_on_name_ccnew ON users (name); UPDATE pg_index SET indisvalid = false WHERE indexrelid = 'public.index_users_on_name_ccnew'::regclass; SQL database = validation::Sources::Database.new(conn) structure_sql = validation::Sources::StructureSql.new(Rails.root.join('db/structure.sql')) [validation::Validators::MissingIndexes, validation::Validators::ExtraIndexes].each do |validator| puts validator.new(structure_sql, database).execute.map(&:error_message).grep(/index_users_on_name/) end raise ActiveRecord::Rollback end -
MissingIndexesreportsindex_users_on_namewith theREINDEXhint, andExtraIndexesreportsindex_users_on_name_ccnew. On master, there is no hint andindex_users_on_name_ccnewisn't reported. -
For the drop hint, add
DROP INDEX public.index_users_on_name; CREATE INDEX index_users_on_name ON users (username);before the firstUPDATEand run it again.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.