Advanced issue search should allow searching in issue comments

Problem to solve

Proposal

Proposing to implement this functionality within the work_items index. For context, the issues index is deprecated and will be removed in a future milestone. The plan is to store all issue notes in fields (one for internal, one for non-internal) in the work items index to allow for searching. Any comment create, update or delete will require indexing the related issue/work item.

Implementation plan

MR 1: Add new fields notes and notes_internal to work items index

  • Create an Advanced search migration using the Elastic::MigrationUpdateMappingsHelper to add two new fields. The fields should have mapping settings: type: :text, index_options: 'positions', analyzer: :code_analyzer
  • In Search::Elastic::Types::WorkItem, add the new fields to base_mappings method

MR 2: Populate new fields in work items index and keep in sync when notes change

  • In Search::Elastic::References::WorkItem, populate the fields in build_indexed_json in Search::Elastic::References::WorkItem. Guard the field population with ::Elastic::DataMigrationService.migration_has_finished? to ensure the migration adding the field has finished. Populate both fields by combining all notes of that type (notes_internal or notes) into one string separated by newlines.
  • Add preloading for notes to the preload_indexing_data scope in WorkItem model
    • In EE::Note model, queue the noteable for indexing by overriding a few methods. The methods should only queue for indexing for notes on issues
      • maintain_elasticsearch_create - when a note is created
      • maintain_elasticsearch_update - when a note is modified (only if note or confidential is changed)
      • maintain_elasticsearch_update - when a note is deleted
  • specs updates

MR 3: Implement search

  • In Search::Elastic::WorkItemQueryBuilder, add the new fields to the searched fields. This should be guarded by ::Elastic::DataMigrationService.migration_has_finished? and new feature flag
  • specs updates

MR 4: Backfilling

  • In Search::Elastic::References::WorkItem, increase the SCHEMA_VERSION constant to a new YY_MM value.
  • Add a new migration using the Elastic:: MigrationReindexBasedOnSchemaVersion to backfill the new fields. I chose this migration helper because we need to backfill fields that may be empty

Release notes

This feature adds the ability to search within issue comments when searching for GitLab issues using advanced search, with results in the issue scope containing issues that match the search term in both issue descriptions and issue comments. This enhancement addresses a usability problem where users previously experienced a fragmented search workflow, having to search issues and comments separately, which made it difficult to efficiently find relevant information and required switching between different search scopes to get a complete picture.

Edited by Changzheng Liu