Keep epics index up to date in Elasticsearch
What does this MR do and why?
Updates Elasticsearch when Epics are changed.
Context
Currently all epic searches are doing a Basic Search and we want to allow Advanced Search to be used when Elasticsearch is available for faster and better searching.
To achieve this, we need the following:
- Create the index in Elasticsearch: Create Epic Elasticsearch index (!121635 - merged)
- Make sure epics are created/updated/deleted when needed:
👈 This MR- Enable the feature flag and remove from code
- Backfill all epics: Elastic migration to backfill epics (!123618 - merged)
- Perform advanced search using the new epic index
Note: this feature is guarded by a feature flag which prevents epics from being added to the index while the second MR is not merged. The first three MRs should be merged in the same milestone.
Details
The following changes prompt changes in the index:
- Epic is created
-
ApplicationVersionedSearchcallsProcessBookkeepingService.track!for the epic
-
- Epic is updated
-
ApplicationVersionedSearchcallsProcessBookkeepingService.track!for the epic
-
- Epic is deleted
-
ApplicationVersionedSearchcallsProcessBookkeepingService.track!for the epic
-
- Epic's inherited date relations change their dates
-
UpdateAssociatedEpicsOnDateChangecallsProcessBookkeepingService.track!for the epic in anafter_updatecallback. Documentation.
-
- Epic is moved to another group
-
ApplicationVersionedSearchcallsProcessBookkeepingService.track!for the epic
-
- Epic's group's
visibility_levelchanges- Calls
ProcessBookkeepingService.maintain_indexed_group_associations!using anafter_update_commitcallback which enqueuesElasticAssociationIndexerWorkerfor the group withepicsas associations.
- Calls
- Epic's group is deleted
- Enqueues
ElasticAssociationDeletionWorkerusing anafter_destroy_commitcallback. The worker deletes epics belonging to the deleted group usingdelete_by_query
- Enqueues
- Epic's group's parent changes
- Enqueues
Elastic::NamespaceUpdateWorkerusing anafter_updatecallback if the parent changed. The worker callsProcessBookkeepingService.maintain_indexed_group_associations!which enqueuesElasticAssociationIndexerWorkerfor the group withepicsas associations.
- Enqueues
- Epic's group is transferred to another group
- Enqueues
Elastic::NamespaceUpdateWorkerusing anafter_updatecallback if the parent changed. The worker callsProcessBookkeepingService.maintain_indexed_group_associations!which enqueuesElasticAssociationIndexerWorkerfor the group withepicsas associations.
- Enqueues
- If namespace index limiting is enabled and the epic's group is moved from an indexed namespace to a non-indexed namespace
- Enqueues
ElasticNamespaceIndexerWorkerwith the:deleteoption. The worker enqueuesElasticAssociationDeletionWorkerworkers for the group and its descendent groups which deletes epics belonging to those groups usingdelete_by_query
- Enqueues
- If namespace index limiting is enabled and the epic's group is moved from a non-indexed namespace to an indexed namespace
- Enqueues
ElasticNamespaceIndexerWorkerwith the:indexoption. The worker callsElastic::ProcessBookkeepingService.maintain_indexed_group_associations!for the group.
- Enqueues
All of these cases are tested in ee/spec/elastic_integration/epic_index_spec.rb.
Note that each one of these operations are behind the feature flag elastic_index_epics.
How to set up and validate locally
- Enable the feature flag:
Feature.enable(:elastic_index_epics). - Ensure the migration for creating the index is complete by running
Elastic::MigrationWorker.new.performa few times. - Perform the cases listed in the Details section and after each step, test that the epic is in the correct in Elasticsearch.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #250699 (closed)