Risk Map for Elasticsearch
Topic to Evaluate
This issue is to describe the facets where Elasticsearch can be evaluated on quality, to document how we are testing for that quality, where we are not explicitly or implicitly testing, and to inform the team as to what areas we can focus more attention on in an ongoing way.
- Should we expand this issue to include basic search facets?
- Should this map exist as an issue or be made into a page in the handbook?
- How do we evaluate the risk level column? Do we even want to keep it? It could be useful in prioritizing test coverage work in the future.
- I'm uncertain how many of these facets are our responsibility, perhaps some of them are the responsibility of the Elasticsearch team, if so, we don't want to duplicate their efforts already.
- Should we only be linking to open issues and MRs, or do we want to have some historical context by linking closed issues and MRs?
- Should the table below be a spreadsheet?
Facets to Evaluate
|Facet||Are we explicitly testing this?||Where? (add link to code)||Are we implicitly testing this?||Where and how?||Issues/MRs related to this facet||Risk level||Impact||Likelihood|
|Redundancy, high availability, failover, disaster recovery||#8010||Moderate||Critical||Rare|
|Shard distribution/skewing||#10519, grab#138||Moderate||Negligible||Likely|
|Indexing, memory use||yes||Performance testing||Moderate||Negligible||Likely|
|Indexing, cpu use||yes||Performance testing||#224629||Moderate||Negligible||Likely|
|Indexing, speed||yes, end to end tests||api tests, elasticsearch_reindexing_spec||#227724 (closed), #231425||High||Marginal||Likely|
|Indexing, accuracy||yes, unit test||project_spec||!29952 (merged), !37834 (merged)||Moderate||Critical||Unlikely|
|API, parity||planned||This issue will test both the UI and API
|#229056 (closed) #222373 (closed)||High||Negligible||Certain|
|Search accuracy, ranking, recall||yes, unit test||snippet_search_results_spec||!36774 (merged), #228575 (closed) stemming, pluralization||Moderate||Negligible||Likely|
|Search, speed||yes||Performance testing||#212798 #229627 #33562 (closed) #229981 #225998 (closed) #243667||High||Marginal||Certain|
|Search, efficiency||#225692 (closed) #34457 (closed)||High||Negligible||Certain|
|Regex, code matching||#225171 (closed)||High||Marginal||Certain|
|Regex, efficiency||#224629, https://gitlab.com/gitlab-org/gitlab/-/issues/228855||High||Marginal||Certain|
|Version compatibility||yes (v6.4.2, and v7.7.0)||.gitlab-ci.yml||Moderate||Negligible||Likely|
|Performance at scale||yes||UI and API||Extreme||Critical||Likely|
|Security, data leaks||https://gitlab.com/gitlab-org/gitlab/-/issues/37959 https://gitlab.com/gitlab-org/gitlab/-/issues/37979 https://gitlab.com/gitlab-org/gitlab/-/issues/238112||High||Marginal||Certain|
|Performance bar / Telemetry||yes, unit test||elasticsearch_spec, global_search_spec, search_counter_spec, search_spec||!36899 (merged) !32111 (merged)||Moderate||Negligible||Unlikely|
|Integrations, AWS credentials||yes, unit test||client_test||gitlab-elasticsearch-indexer!82 (merged)||Moderate||Marginal||Possible|
|Operation cost, storage||High||Marginal||Likely|
|Operation cost, traffic||#224472 (closed)||High||Marginal||Certain|
Performance testing: https://gitlab.com/gitlab-org/quality/performance/-/blob/master/k6/tests/api/api_v4_search_global.js https://gitlab.com/gitlab-org/quality/performance/-/blob/master/k6/tests/api/api_v4_search_groups.js https://gitlab.com/gitlab-org/quality/performance/-/blob/master/k6/tests/api/api_v4_search_projects.js
Determine feasibility of the feature
Create issue for implementation or update existing implementation issue description with implementation proposal
Set weight on implementation issue
If weight is greater than 5, break issue into smaller issues
Risks and Implementation Considerations
Add feature and the corresponding
Ping the PM and EM.