Create Epic Elasticsearch index
What does this MR do and why?
Creates an index for Epics in Elasticsearch.
Currently all epic searches are doing a Basic Search and we want to allow Advanced Search to be used when Elasticsearch is available for faster and better searching.
To achieve this, we need the following:
- Create the index in Elasticsearch
This MR - Make sure epics are created/updated/deleted when needed: Keep epics index up to date in Elasticsearch (!123526 - merged)
- Enable the feature flag and remove from code
- Backfill all epics: Elastic migration to backfill epics (!123618 - merged)
- Perform advanced search using the new epic index
Note: this feature is guarded by a feature flag which prevents epics from being added to the index while the second MR is not merged. The first three MRs should be merged in the same milestone.
Epic index settings and mappings:
Click to expand
{
"gitlab-development-epics-20230613-1336" : {
"aliases" : {
"gitlab-development-epics" : { }
},
"mappings" : {
"dynamic" : "strict",
"_meta" : {
"created_by" : "16.1.0-pre"
},
"properties" : {
"author_id" : {
"type" : "integer"
},
"confidential" : {
"type" : "boolean"
},
"created_at" : {
"type" : "date"
},
"description" : {
"type" : "text"
},
"due_date" : {
"type" : "date"
},
"group_id" : {
"type" : "integer"
},
"hashed_root_namespace_id" : {
"type" : "integer"
},
"id" : {
"type" : "integer"
},
"iid" : {
"type" : "integer"
},
"label_ids" : {
"type" : "keyword"
},
"schema_version" : {
"type" : "short"
},
"start_date" : {
"type" : "date"
},
"state" : {
"type" : "keyword"
},
"title" : {
"type" : "text"
},
"traversal_ids" : {
"type" : "keyword"
},
"type" : {
"type" : "keyword"
},
"updated_at" : {
"type" : "date"
}
}
},
"settings" : {
"index" : {
"codec" : "best_compression",
"highlight" : {
"max_analyzed_offset" : "1048576"
},
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "5",
"provided_name" : "gitlab-development-epics-20230613-1336",
"creation_date" : "1686663417552",
"analysis" : {
"filter" : {
"word_delimiter_graph_filter" : {
"type" : "word_delimiter_graph",
"preserve_original" : "true"
}
},
"normalizer" : {
"sha_normalizer" : {
"filter" : [
"lowercase"
],
"type" : "custom"
}
},
"analyzer" : {
"email_analyzer" : {
"tokenizer" : "email_tokenizer"
},
"default" : {
"filter" : [
"lowercase",
"stemmer"
],
"tokenizer" : "standard"
},
"whitespace_reverse" : {
"filter" : [
"lowercase",
"asciifolding",
"reverse"
],
"tokenizer" : "whitespace"
},
"path_analyzer" : {
"filter" : [
"lowercase",
"asciifolding"
],
"type" : "custom",
"tokenizer" : "path_tokenizer"
},
"code_analyzer" : {
"filter" : [
"word_delimiter_graph_filter",
"flatten_graph",
"lowercase",
"asciifolding",
"remove_duplicates"
],
"type" : "custom",
"tokenizer" : "whitespace"
},
"my_ngram_analyzer" : {
"filter" : [
"lowercase"
],
"tokenizer" : "my_ngram_tokenizer"
}
},
"tokenizer" : {
"my_ngram_tokenizer" : {
"token_chars" : [
"letter",
"digit"
],
"min_gram" : "2",
"type" : "ngram",
"max_gram" : "3"
},
"email_tokenizer" : {
"type" : "uax_url_email"
},
"path_tokenizer" : {
"reverse" : "true",
"type" : "path_hierarchy"
}
}
},
"number_of_replicas" : "1",
"uuid" : "UoNgar5CRkOiFgU7fedELg",
"version" : {
"created" : "8060299"
}
}
}
}
}
Routing is done by root ancestor group so that we leverage traversal_ids
and limit documents to epics in the ancestor and its descendent groups.
Example document
{
"_index" : "gitlab-development-epics-20230628-0730",
"_id" : "epic_56",
"_routing" : "group_78",
"_source" : {
"id" : 56,
"iid" : 1,
"group_id" : 79,
"created_at" : "2023-06-28T07:27:44.205Z",
"updated_at" : "2023-06-28T07:27:44.205Z",
"title" : "Some epic",
"description" : "description",
"state" : "opened",
"confidential" : false,
"author_id" : 1,
"label_ids" : [ ],
"start_date" : null,
"due_date" : null,
"traversal_ids" : "78-79-",
"hashed_root_namespace_id" : 368,
"visibility_level" : 0,
"schema_version" : 2306,
"type" : "epic"
}
}
Logs
Creating the index:
"Elastic::MigrationWorker","message":"MigrationWorker: migration[CreateEpicIndex] executing migrate method"
"CreateEpicIndex","message":"[Elastic::Migration: 20230518135700] Creating standalone epic index gitlab-development-epics"
"Elastic::MigrationWorker","message":"MigrationWorker: migration[CreateEpicIndex] updating with completed: true"
How to set up and validate locally
- Check that an epic index doesn't exist:
curl "http://localhost:9200/_cat/aliases/gitlab-development-epics?h=i"
- Execute the migration worker a few times:
Elastic::MigrationWorker.new.perform
- Optional: view the logs:
tail -f log/elasticsearch.log
- Check that the epic index now exists:
curl "http://localhost:9200/_cat/aliases/gitlab-development-epics?h=i"
- Disable the feature flag:
Feature.disable(:elastic_index_epics)
- Update an epic, e.g.
Epic.first.update(title: "test")
- Notice that the epic isn't scheduled to be updated in Elasticsearch (no logs in
elasticsearch.log
) - Enable the feature flag:
Feature.enable(:elastic_index_epics)
- Update an epic, e.g.
Epic.first.update(title: "test2")
- Notice that the epic is scheduled to be updated in Elasticsearch.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #250699 (closed)
Merge request reports
Activity
changed milestone to %16.1
added Actionable InsightExploration needed Category:Global Search Category:Portfolio Management Enterprise Edition GitLab Starter Next1-3 releases UX advanced search backend customer devopsdata stores direction epics groupglobal search internal customer sectioncore platform typefeature workflowin dev labels
assigned to @maddievn
removed Actionable InsightExploration needed label
removed Next1-3 releases label
1 Warning This merge request is quite big (549 lines changed), please consider splitting it into multiple merge requests. Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Valery Sizov (
@vsizov
) (UTC+1, 1 hour behind@maddievn
)James Fargher (
@proglottis
) (UTC+12, 10 hours ahead of@maddievn
)~"Global Search" Reviewer review is optional for ~"Global Search" John Mason (
@johnmason
) (UTC-4, 6 hours behind@maddievn
)To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
DangerAllure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for c802c0eaexpand test summary
+-----------------------------------------------------------------------+ | suites summary | +------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------+--------+--------+---------+-------+-------+--------+ | Create | 6 | 2 | 1 | 8 | 9 | ❌ | | Monitor | 4 | 0 | 0 | 4 | 4 | ❗ | | Framework sanity | 0 | 0 | 1 | 0 | 1 | ➖ | | Plan | 4 | 0 | 0 | 4 | 4 | ❗ | | Data Stores | 2 | 0 | 0 | 0 | 2 | ✅ | | Manage | 1 | 0 | 0 | 0 | 1 | ✅ | | Govern | 2 | 0 | 0 | 0 | 2 | ✅ | +------------------+--------+--------+---------+-------+-------+--------+ | Total | 19 | 2 | 2 | 16 | 23 | ❌ | +------------------+--------+--------+---------+-------+-------+--------+
e2e-package-and-test:
test report for c802c0eaexpand test summary
+------------------------------------------------------------+ | suites summary | +-------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------+--------+--------+---------+-------+-------+--------+ +-------+--------+--------+---------+-------+-------+--------+ | Total | 0 | 0 | 0 | 0 | 0 | ➖ | +-------+--------+--------+---------+-------+-------+--------+
- A deleted user
added feature flag label
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@349387c4
added 2 commits
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@a6fb20a1
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@8b010f84
added 268 commits
-
3d5a062d...823d040d - 258 commits from branch
master
- feba03db - Create Epic index
- 3821338f - Add epic instance proxy and config
- 5baca4f8 - Add class proxy and N+1 specs
- a6dc46c3 - [skip-ci] Move rubocop disables to todos
- acfa2ec2 - Add rake task to index epics
- 34c2b391 - Update use_elasticsearch?
- a39d8115 - Use maintain elasticsearch
- f970c921 - Add search curation feature flag for epics
- 36df5289 - Partition by root namespace
- d5a5bcf1 - Route by group id
Toggle commit list-
3d5a062d...823d040d - 258 commits from branch
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@fd28bf38
added 256 commits
-
d5a5bcf1...72d85a10 - 246 commits from branch
master
- 7f430a67 - Create Epic index
- 7f5002a7 - Add epic instance proxy and config
- add8c182 - Add class proxy and N+1 specs
- a090ac52 - [skip-ci] Move rubocop disables to todos
- a561125d - Add rake task to index epics
- a55b6449 - Update use_elasticsearch?
- 4a841533 - Use maintain elasticsearch
- 49f5fdb4 - Add search curation feature flag for epics
- 7f1e5ca3 - Partition by root namespace
- 40a18815 - Route by group id
Toggle commit list-
d5a5bcf1...72d85a10 - 246 commits from branch
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@5b8132ea
mentioned in merge request !122183 (merged)
added 1 commit
- c9a0f483 - Maintain elasticsearch when migraiton is complete
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@f8b384bc
added 422 commits
-
c9a0f483...8bd5f5b7 - 421 commits from branch
master
- 8ca58981 - Create Epic index
-
c9a0f483...8bd5f5b7 - 421 commits from branch
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@079e83d9
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@8169d965
added 869 commits
-
2b25d077...b57fa00f - 867 commits from branch
master
- e2914e81 - Create Epic index
- 31fdc30a - Add feature type spec
-
2b25d077...b57fa00f - 867 commits from branch
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@6e3b86db
added 2 commits
- Resolved by Madelein van Niekerk
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@70d7be54
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@6eebf0c3
added 1 commit
- 2a74b422 - Remove spec for testing epics moving to another group
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@1f337d8a
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@fea00fda
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@3742cffc
added 705 commits
-
4e73df53...f79f8121 - 704 commits from branch
master
- 525ec3f6 - Create Epic index
-
4e73df53...f79f8121 - 704 commits from branch
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@96d66c10
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@f6a8d51c