<!-- Instructions: Use this template for a proof of concept or when a deeper technical evaluation is required. Please weigh tech evaluation issues and follow the instructions below accordingly. --> ### Topic to Evaluate This issue is to describe the facets where Elasticsearch can be evaluated on quality, to document how we are testing for that quality, where we are not explicitly or implicitly testing, and to inform the team as to what areas we can focus more attention on in an ongoing way. Some questions: 1. Should we expand this issue to include basic search facets? 2. Should this map exist as an issue or be made into a page in the handbook? 3. How do we evaluate the risk level column? Do we even want to keep it? It could be useful in prioritizing test coverage work in the future. 4. I'm uncertain how many of these facets are our responsibility, perhaps some of them are the responsibility of the Elasticsearch team, if so, we don't want to duplicate their efforts already. 5. Should we only be linking to open issues and MRs, or do we want to have some historical context by linking closed issues and MRs? 6. Should the table below be a spreadsheet? <!-- Describe the related issue and challenge we need to establish a proof of concept for--> ### Facets to Evaluate <!-- Outline the tasks with issues that you need to evaluate as a part of the implementation issue --> | Facet | Are we explicitly testing this? | Where? (add link to code) | Are we implicitly testing this? | Where and how? | Issues/MRs related to this facet | Risk level | Impact | Likelihood | | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | Redundancy, high availability, failover, disaster recovery | | | | | https://gitlab.com/gitlab-org/gitlab/-/issues/8010 | Moderate | Critical | Rare | | Shard distribution/skewing | | | | | https://gitlab.com/gitlab-org/gitlab/-/issues/10519, [grab#138](https://gitlab.com/gitlab-com/account-management/apac/singapore/grab/-/issues/138) | Moderate | Negligible | Likely | | Indexing, memory use | | | yes | [Performance testing](https://gitlab.com/gitlab-org/quality/performance/-/tree/master/k6/tests/web) | | Moderate | Negligible | Likely | | Indexing, cpu use | | | yes | [Performance testing](https://gitlab.com/gitlab-org/quality/performance/-/tree/master/k6/tests/web) | https://gitlab.com/gitlab-org/gitlab/-/issues/224629 | Moderate | Negligible | Likely | | Indexing, speed | | | yes, end to end tests | [api tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/api/enablement/elasticsearch), [elasticsearch_reindexing_spec](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/enablement/elasticsearch/elasticsearch_reindexing_spec.rb) | https://gitlab.com/gitlab-org/gitlab/-/issues/227724, https://gitlab.com/gitlab-org/gitlab/-/issues/231425| High | Marginal | Likely | | Indexing, accuracy | yes, unit test |[project_spec](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/models/project_spec.rb#L1692) | | | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29952, https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37834 | Moderate | Critical | Unlikely | | API, parity | planned | This issue will test both the UI and API <br> https://gitlab.com/gitlab-org/gitlab/-/issues/212390 | | | https://gitlab.com/gitlab-org/gitlab/-/issues/229056 https://gitlab.com/gitlab-org/gitlab/-/issues/222373| High | Negligible | Certain | | ~"api:consistency" | | | | | | Moderate | Negligible | Likely | | Search accuracy, ranking, recall | yes, unit test | [snippet_search_results_spec](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/lib/gitlab/elastic/snippet_search_results_spec.rb#L24)| | | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36774, https://gitlab.com/gitlab-org/gitlab/-/issues/228575 [stemming, pluralization](https://gitlab.com/gitlab-org/gitlab/-/issues/277390) | Moderate | Negligible | Likely | | Search, speed | yes | [Performance testing](https://gitlab.com/gitlab-org/quality/performance/-/tree/master/k6/tests/web) | | | https://gitlab.com/gitlab-org/gitlab/-/issues/212798 https://gitlab.com/gitlab-org/gitlab/-/issues/229627 https://gitlab.com/gitlab-org/gitlab/-/issues/33562 https://gitlab.com/gitlab-org/gitlab/-/issues/229981 https://gitlab.com/gitlab-org/gitlab/-/issues/225998 https://gitlab.com/gitlab-org/gitlab/-/issues/243667 | High | Marginal | Certain | | Search, efficiency | | | | | https://gitlab.com/gitlab-org/gitlab/-/issues/225692 https://gitlab.com/gitlab-org/gitlab/-/issues/34457 | High | Negligible | Certain | | Regex, code matching | | | | | https://gitlab.com/gitlab-org/gitlab/-/issues/225171 | High | Marginal | Certain | | Regex, efficiency | | | | | https://gitlab.com/gitlab-org/gitlab/-/issues/224629, https://gitlab.com/gitlab-org/gitlab/-/issues/228855 | High | Marginal | Certain | | Version compatibility | yes (v6.4.2, and v7.7.0) | [.gitlab-ci.yml](https://gitlab.com/gitlab-org/quality/nightly/-/blob/master/.gitlab-ci.yml) | | | | Moderate | Negligible | Likely | | Performance at scale | yes |[UI](https://gitlab.com/gitlab-org/quality/performance/-/tree/master/k6/tests/web) and [API](https://gitlab.com/gitlab-org/quality/performance/-/tree/master/k6/tests/api) | | | | Extreme | Critical | Likely | | Security, data leaks | | | | | https://gitlab.com/gitlab-org/gitlab/-/issues/37959 https://gitlab.com/gitlab-org/gitlab/-/issues/37979 https://gitlab.com/gitlab-org/gitlab/-/issues/238112| High | Marginal | Certain | | Performance bar / Telemetry | yes, unit test | [elasticsearch_spec](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/lib/peek/views/elasticsearch_spec.rb#L18), [global_search_spec](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/features/global_search_spec.rb#L16), [search_counter_spec](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/usage_data_counters/search_counter_spec.rb#L7), [search_spec](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/requests/api/search_spec.rb#L87)| | | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36899 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/32111 | Moderate | Negligible | Unlikely | | Integrations, AWS credentials | yes, unit test | [client_test](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/blob/master/elastic/client_test.go#L89)| | | https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/merge_requests/82 | Moderate | Marginal | Possible | | Operation cost, storage | | | | | | High | Marginal | Likely | | Operation cost, traffic | | | | | https://gitlab.com/gitlab-org/gitlab/-/issues/224472 | High | Marginal | Certain | Performance testing: https://gitlab.com/gitlab-org/quality/performance/-/blob/master/k6/tests/api/api_v4_search_global.js https://gitlab.com/gitlab-org/quality/performance/-/blob/master/k6/tests/api/api_v4_search_groups.js https://gitlab.com/gitlab-org/quality/performance/-/blob/master/k6/tests/api/api_v4_search_projects.js - [ ] Determine feasibility of the feature - [ ] Create issue for implementation or update existing implementation issue description with implementation proposal - [ ] Set weight on implementation issue - [ ] If weight is greater than 5, break issue into smaller issues - [ ] Add task - [ ] Add task ### Risks and Implementation Considerations <!-- Identify any risks found in the research, whether this is performance, impacts to other functionality or other bugs --> ### Team - [ ] Add ~"workflow::planning breakdown" ~feature and the corresponding `~devops::<stage>` and `~group::<group>` labels. - [ ] Ping the PM and EM.