Skip to content

Keep epics index up to date in Elasticsearch

Madelein van Niekerk requested to merge 250699-keep-epics-up-to-date into master

What does this MR do and why?

Updates Elasticsearch when Epics are changed.

Context

Currently all epic searches are doing a Basic Search and we want to allow Advanced Search to be used when Elasticsearch is available for faster and better searching.

To achieve this, we need the following:

Note: this feature is guarded by a feature flag which prevents epics from being added to the index while the second MR is not merged. The first three MRs should be merged in the same milestone.

Details

The following changes prompt changes in the index:

  • Epic is created
    • ApplicationVersionedSearch calls ProcessBookkeepingService.track! for the epic
  • Epic is updated
    • ApplicationVersionedSearch calls ProcessBookkeepingService.track! for the epic
  • Epic is deleted
    • ApplicationVersionedSearch calls ProcessBookkeepingService.track! for the epic
  • Epic's inherited date relations change their dates
    • UpdateAssociatedEpicsOnDateChange calls ProcessBookkeepingService.track! for the epic in an after_update callback. Documentation.
  • Epic is moved to another group
    • ApplicationVersionedSearch calls ProcessBookkeepingService.track! for the epic
  • Epic's group's visibility_level changes
    • Calls ProcessBookkeepingService.maintain_indexed_group_associations! using an after_update_commit callback which enqueues ElasticAssociationIndexerWorker for the group with epics as associations.
  • Epic's group is deleted
    • Enqueues ElasticAssociationDeletionWorker using an after_destroy_commit callback. The worker deletes epics belonging to the deleted group using delete_by_query
  • Epic's group's parent changes
    • Enqueues Elastic::NamespaceUpdateWorker using an after_update callback if the parent changed. The worker calls ProcessBookkeepingService.maintain_indexed_group_associations! which enqueues ElasticAssociationIndexerWorker for the group with epics as associations.
  • Epic's group is transferred to another group
    • Enqueues Elastic::NamespaceUpdateWorker using an after_update callback if the parent changed. The worker calls ProcessBookkeepingService.maintain_indexed_group_associations! which enqueues ElasticAssociationIndexerWorker for the group with epics as associations.
  • If namespace index limiting is enabled and the epic's group is moved from an indexed namespace to a non-indexed namespace
    • Enqueues ElasticNamespaceIndexerWorker with the :delete option. The worker enqueues ElasticAssociationDeletionWorker workers for the group and its descendent groups which deletes epics belonging to those groups using delete_by_query
  • If namespace index limiting is enabled and the epic's group is moved from a non-indexed namespace to an indexed namespace
    • Enqueues ElasticNamespaceIndexerWorker with the :index option. The worker calls Elastic::ProcessBookkeepingService.maintain_indexed_group_associations! for the group.

All of these cases are tested in ee/spec/elastic_integration/epic_index_spec.rb.

Note that each one of these operations are behind the feature flag elastic_index_epics.

How to set up and validate locally

  1. Enable the feature flag: Feature.enable(:elastic_index_epics).
  2. Ensure the migration for creating the index is complete by running Elastic::MigrationWorker.new.perform a few times.
  3. Perform the cases listed in the Details section and after each step, test that the epic is in the correct in Elasticsearch.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #250699 (closed)

Edited by Madelein van Niekerk

Merge request reports

Loading