Index group milestones and keep them in sync with group changes

What

Group milestones are indexed in Elasticsearch and kept in sync with their group. This ships in one MR, !257456 (merged), so no group milestone enters the index before the sync code exists.

Changes

  • The group milestone Elasticsearch document includes the group's namespace_id, traversal_ids, visibility_level, and archived.
  • Group milestone documents are routed by n_<root_ancestor_id>, based on the group's traversal ids. Project milestones keep their project_<id> routing.
  • Milestone#use_elasticsearch? is overridden so project milestones use project.use_elasticsearch? and group milestones use group.use_elasticsearch?. Before, group milestones fell back to project&.use_elasticsearch?, which is nil, so they were never indexed.
  • Group milestone documents are removed when a group is deleted, transferred to a new root, or removed from limited indexing (Search::ElasticGroupAssociationDeletionWorker).
  • A group's milestones are reindexed:
    • when its visibility changes or the group is newly enabled for indexing: Elastic::ProcessBookkeepingService.maintain_indexed_namespace_associations! now includes milestones, the same way it already includes epics.
    • when it's transferred, along with its subgroups' milestones: EE::Groups::TransferService.
    • when it's archived or unarchived, along with its subgroups' milestones: Search::Elastic::GroupArchivedEventWorker.
  • gitlab:elastic:index_group_entities indexes group milestones; gitlab:elastic:index already covers them through ElasticNamespaceIndexerWorker. Group import (direct transfer and file-based) indexes them through the normal create callback.

Group milestones created before this MR is deployed have no documents yet. Backfilling them is a separate issue, #630857, that ships once this MR is deployed.

Why indexing and sync must ship together

milestones.group_id is ON DELETE CASCADE, so a deleted group's milestone rows vanish without callbacks; if those milestones were already indexed, their documents would be orphaned forever. A group transfer changes its routing (n_<root_ancestor_id>), so the next milestone update would write a second copy under the new routing while the old copy stays behind on the old shard. Turning on indexing before this sync code exists would leave both cases unrecoverable.

Notes

#630618 still filters group milestone documents out of search results until it lands, so group milestones aren't searchable until then.

Part of #428589. Based on the implementation plan.

Edited by Ravi Kumar