Remove join_field from main Advanced Search index documents

What does this MR do and why?

Adds an Advanced Search migration, RemoveJoinFieldFromMainIndex, that strips the unused join_field from stored documents in the main index. This is task 4 of #630682.

Nothing reads join_field anymore (!259063 (merged)), and Rails no longer writes it (!259064 (merged)). The main index mapping is dynamic: 'strict', so when the reindex in #600600 drops the field from the mapping, any document that still has it fails to copy and the reindex aborts. This migration removes the field from those documents first.

Merge only after the indexer bump. gitlab-elasticsearch-indexer stops writing join_field on blobs in gitlab-elasticsearch-indexer!947 (merged), which ships in v5.14.18 (gitlab-elasticsearch-indexer!960 (closed)). Master still runs v5.14.17. If this migration runs before GITLAB_ELASTICSEARCH_INDEXER_VERSION is 5.14.18, every blob the indexer rewrites afterwards gets join_field back.

Advanced Search migration

• batched!, batch_size 10_000, throttle_delay 1.minute, retry_on_failure.
• Each batch runs update_by_query on the main index alias with an exists query on join_field and ctx._source.remove('join_field').
• completed? refreshes the index and counts documents that still have join_field. The migration is done when the count is 0.
• All document types. It doesn't filter by type, because join_field is on both blob and milestone documents.
• Why not MigrationRemoveFieldsHelper. The helper filters by one document type, which it builds from a model class (DOCUMENT_TYPE.name.underscore). Blob documents have no model class, so it can't target them. Its update_by_query also has no conflicts: 'proceed'. Making it work would mean overriding query, completed? and update_by_query!, which is nearly the whole helper.
• Version conflicts. It uses conflicts: 'proceed'. The indexer rewrites blobs while this runs, and a conflicting document is being reindexed anyway. Any document still left is picked up by a later batch, because completed? counts what remains.
• Routing. update_by_query keeps each document's routing (project_<id>). The spec checks that milestones can still be fetched with their project routing afterwards.
• Reversibility. It only removes a field that nothing reads. It deletes no documents, and it's safe to run again.

Sizing

Locally, join_field is on 18,147 blob and 85 milestone documents (all of them, out of 18,253 in the main index). The production count hasn't been taken yet. That's also task 5 of #630682. At 10,000 documents a minute, 10 million documents take about 17 hours. If production has far more blob documents, the batch size can go up before merge.

References

• #630682
• Blocks #600600

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

1. Check how many main index documents have join_field:

   curl -s "localhost:9200/gitlab-development/_count" -H 'Content-Type: application/json' \
     -d '{"query":{"exists":{"field":"join_field"}}}'

2. In a Rails console, load the migration (files in ee/elastic/migrate aren't autoloaded) and run it until it finishes:

   require Rails.root.join('ee/elastic/migrate/20261013143426_remove_join_field_from_main_index.rb').to_s
   
   migration = RemoveJoinFieldFromMainIndex.new(20261013143426)
   migration.migrate until migration.completed?

3. Run the count again. It returns 0.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code