Keep epics index up to date in Elasticsearch
What does this MR do and why?
Updates Elasticsearch when Epics are changed.
Context
Currently all epic searches are doing a Basic Search and we want to allow Advanced Search to be used when Elasticsearch is available for faster and better searching.
To achieve this, we need the following:
- Create the index in Elasticsearch: Create Epic Elasticsearch index (!121635 - merged)
- Make sure epics are created/updated/deleted when needed:
👈 This MR- Enable the feature flag and remove from code
- Backfill all epics: Elastic migration to backfill epics (!123618 - merged)
- Perform advanced search using the new epic index
Note: this feature is guarded by a feature flag which prevents epics from being added to the index while the second MR is not merged. The first three MRs should be merged in the same milestone.
Details
The following changes prompt changes in the index:
- Epic is created
-
ApplicationVersionedSearch
callsProcessBookkeepingService.track!
for the epic
-
- Epic is updated
-
ApplicationVersionedSearch
callsProcessBookkeepingService.track!
for the epic
-
- Epic is deleted
-
ApplicationVersionedSearch
callsProcessBookkeepingService.track!
for the epic
-
- Epic's inherited date relations change their dates
-
UpdateAssociatedEpicsOnDateChange
callsProcessBookkeepingService.track!
for the epic in anafter_update
callback. Documentation.
-
- Epic is moved to another group
-
ApplicationVersionedSearch
callsProcessBookkeepingService.track!
for the epic
-
- Epic's group's
visibility_level
changes- Calls
ProcessBookkeepingService.maintain_indexed_group_associations!
using anafter_update_commit
callback which enqueuesElasticAssociationIndexerWorker
for the group withepics
as associations.
- Calls
- Epic's group is deleted
- Enqueues
ElasticAssociationDeletionWorker
using anafter_destroy_commit
callback. The worker deletes epics belonging to the deleted group usingdelete_by_query
- Enqueues
- Epic's group's parent changes
- Enqueues
Elastic::NamespaceUpdateWorker
using anafter_update
callback if the parent changed. The worker callsProcessBookkeepingService.maintain_indexed_group_associations!
which enqueuesElasticAssociationIndexerWorker
for the group withepics
as associations.
- Enqueues
- Epic's group is transferred to another group
- Enqueues
Elastic::NamespaceUpdateWorker
using anafter_update
callback if the parent changed. The worker callsProcessBookkeepingService.maintain_indexed_group_associations!
which enqueuesElasticAssociationIndexerWorker
for the group withepics
as associations.
- Enqueues
- If namespace index limiting is enabled and the epic's group is moved from an indexed namespace to a non-indexed namespace
- Enqueues
ElasticNamespaceIndexerWorker
with the:delete
option. The worker enqueuesElasticAssociationDeletionWorker
workers for the group and its descendent groups which deletes epics belonging to those groups usingdelete_by_query
- Enqueues
- If namespace index limiting is enabled and the epic's group is moved from a non-indexed namespace to an indexed namespace
- Enqueues
ElasticNamespaceIndexerWorker
with the:index
option. The worker callsElastic::ProcessBookkeepingService.maintain_indexed_group_associations!
for the group.
- Enqueues
All of these cases are tested in ee/spec/elastic_integration/epic_index_spec.rb
.
Note that each one of these operations are behind the feature flag elastic_index_epics
.
How to set up and validate locally
- Enable the feature flag:
Feature.enable(:elastic_index_epics)
. - Ensure the migration for creating the index is complete by running
Elastic::MigrationWorker.new.perform
a few times. - Perform the cases listed in the Details section and after each step, test that the epic is in the correct in Elasticsearch.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #250699 (closed)