Create Epic Elasticsearch index
-
Review changes -
-
Download -
Patches
-
Plain diff
What does this MR do and why?
Creates an index for Epics in Elasticsearch.
Currently all epic searches are doing a Basic Search and we want to allow Advanced Search to be used when Elasticsearch is available for faster and better searching.
To achieve this, we need the following:
- Create the index in Elasticsearch
This MR - Make sure epics are created/updated/deleted when needed: Keep epics index up to date in Elasticsearch (!123526 - merged)
- Enable the feature flag and remove from code
- Backfill all epics: Elastic migration to backfill epics (!123618 - merged)
- Perform advanced search using the new epic index
Note: this feature is guarded by a feature flag which prevents epics from being added to the index while the second MR is not merged. The first three MRs should be merged in the same milestone.
Epic index settings and mappings:
Click to expand
{
"gitlab-development-epics-20230613-1336" : {
"aliases" : {
"gitlab-development-epics" : { }
},
"mappings" : {
"dynamic" : "strict",
"_meta" : {
"created_by" : "16.1.0-pre"
},
"properties" : {
"author_id" : {
"type" : "integer"
},
"confidential" : {
"type" : "boolean"
},
"created_at" : {
"type" : "date"
},
"description" : {
"type" : "text"
},
"due_date" : {
"type" : "date"
},
"group_id" : {
"type" : "integer"
},
"hashed_root_namespace_id" : {
"type" : "integer"
},
"id" : {
"type" : "integer"
},
"iid" : {
"type" : "integer"
},
"label_ids" : {
"type" : "keyword"
},
"schema_version" : {
"type" : "short"
},
"start_date" : {
"type" : "date"
},
"state" : {
"type" : "keyword"
},
"title" : {
"type" : "text"
},
"traversal_ids" : {
"type" : "keyword"
},
"type" : {
"type" : "keyword"
},
"updated_at" : {
"type" : "date"
}
}
},
"settings" : {
"index" : {
"codec" : "best_compression",
"highlight" : {
"max_analyzed_offset" : "1048576"
},
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "5",
"provided_name" : "gitlab-development-epics-20230613-1336",
"creation_date" : "1686663417552",
"analysis" : {
"filter" : {
"word_delimiter_graph_filter" : {
"type" : "word_delimiter_graph",
"preserve_original" : "true"
}
},
"normalizer" : {
"sha_normalizer" : {
"filter" : [
"lowercase"
],
"type" : "custom"
}
},
"analyzer" : {
"email_analyzer" : {
"tokenizer" : "email_tokenizer"
},
"default" : {
"filter" : [
"lowercase",
"stemmer"
],
"tokenizer" : "standard"
},
"whitespace_reverse" : {
"filter" : [
"lowercase",
"asciifolding",
"reverse"
],
"tokenizer" : "whitespace"
},
"path_analyzer" : {
"filter" : [
"lowercase",
"asciifolding"
],
"type" : "custom",
"tokenizer" : "path_tokenizer"
},
"code_analyzer" : {
"filter" : [
"word_delimiter_graph_filter",
"flatten_graph",
"lowercase",
"asciifolding",
"remove_duplicates"
],
"type" : "custom",
"tokenizer" : "whitespace"
},
"my_ngram_analyzer" : {
"filter" : [
"lowercase"
],
"tokenizer" : "my_ngram_tokenizer"
}
},
"tokenizer" : {
"my_ngram_tokenizer" : {
"token_chars" : [
"letter",
"digit"
],
"min_gram" : "2",
"type" : "ngram",
"max_gram" : "3"
},
"email_tokenizer" : {
"type" : "uax_url_email"
},
"path_tokenizer" : {
"reverse" : "true",
"type" : "path_hierarchy"
}
}
},
"number_of_replicas" : "1",
"uuid" : "UoNgar5CRkOiFgU7fedELg",
"version" : {
"created" : "8060299"
}
}
}
}
}
Routing is done by root ancestor group so that we leverage traversal_ids
and limit documents to epics in the ancestor and its descendent groups.
Example document
{
"_index" : "gitlab-development-epics-20230628-0730",
"_id" : "epic_56",
"_routing" : "group_78",
"_source" : {
"id" : 56,
"iid" : 1,
"group_id" : 79,
"created_at" : "2023-06-28T07:27:44.205Z",
"updated_at" : "2023-06-28T07:27:44.205Z",
"title" : "Some epic",
"description" : "description",
"state" : "opened",
"confidential" : false,
"author_id" : 1,
"label_ids" : [ ],
"start_date" : null,
"due_date" : null,
"traversal_ids" : "78-79-",
"hashed_root_namespace_id" : 368,
"visibility_level" : 0,
"schema_version" : 2306,
"type" : "epic"
}
}
Logs
Creating the index:
"Elastic::MigrationWorker","message":"MigrationWorker: migration[CreateEpicIndex] executing migrate method"
"CreateEpicIndex","message":"[Elastic::Migration: 20230518135700] Creating standalone epic index gitlab-development-epics"
"Elastic::MigrationWorker","message":"MigrationWorker: migration[CreateEpicIndex] updating with completed: true"
How to set up and validate locally
- Check that an epic index doesn't exist:
curl "http://localhost:9200/_cat/aliases/gitlab-development-epics?h=i"
- Execute the migration worker a few times:
Elastic::MigrationWorker.new.perform
- Optional: view the logs:
tail -f log/elasticsearch.log
- Check that the epic index now exists:
curl "http://localhost:9200/_cat/aliases/gitlab-development-epics?h=i"
- Disable the feature flag:
Feature.disable(:elastic_index_epics)
- Update an epic, e.g.
Epic.first.update(title: "test")
- Notice that the epic isn't scheduled to be updated in Elasticsearch (no logs in
elasticsearch.log
) - Enable the feature flag:
Feature.enable(:elastic_index_epics)
- Update an epic, e.g.
Epic.first.update(title: "test2")
- Notice that the epic is scheduled to be updated in Elasticsearch.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #250699 (closed)
Merge request reports
- version 34eaeba171
- version 333c665086
- version 3229e9a9ba
- version 31e0009fee
- version 30410a9fcf
- version 29b21f4a26
- version 28bf7c0e94
- version 2785b7bcb3
- version 26a65e547a
- version 259758e240
- version 24e4b39690
- version 23525ec3f6
- version 224e73df53
- version 2147918b9e
- version 202a74b422
- version 19ab7e3941
- version 181c68cb26
- version 1731fdc30a
- version 162b25d077
- version 158ca58981
- version 14c9a0f483
- version 1340a18815
- version 12d5a5bcf1
- version 113d5a062d
- version 103d5a062d
- version 93d5a062d
- version 837b734af
- version 7ecaf5903
- version 6b9209b3f
- version 53f233b92
- version 4f221c960
- version 365a5a4c8
- version 21a3e281d
- version 1a5e654a0
- master (base)
- latest versionc802c0ea1 commit,
- version 34eaeba17111 commits,
- version 333c66508610 commits,
- version 3229e9a9ba9 commits,
- version 31e0009fee8 commits,
- version 30410a9fcf8 commits,
- version 29b21f4a267 commits,
- version 28bf7c0e946 commits,
- version 2785b7bcb35 commits,
- version 26a65e547a4 commits,
- version 259758e2403 commits,
- version 24e4b396902 commits,
- version 23525ec3f61 commit,
- version 224e73df538 commits,
- version 2147918b9e7 commits,
- version 202a74b4226 commits,
- version 19ab7e39415 commits,
- version 181c68cb264 commits,
- version 1731fdc30a2 commits,
- version 162b25d0772 commits,
- version 158ca589811 commit,
- version 14c9a0f48311 commits,
- version 1340a1881510 commits,
- version 12d5a5bcf110 commits,
- version 113d5a062d11 commits,
- version 103d5a062d121 commits,
- version 93d5a062d11 commits,
- version 837b734af10 commits,
- version 7ecaf59038 commits,
- version 6b9209b3f7 commits,
- version 53f233b926 commits,
- version 4f221c9605 commits,
- version 365a5a4c85 commits,
- version 21a3e281d4 commits,
- version 1a5e654a03 commits,
- Side-by-side
- Inline