There have been many conversations regarding the possible use of ElasticSearch as a possible mechanism to resolve some of our tougher performance problems with relation to Vulnerability Reports. ElasticSearch appears to have the potential to provide for some of the more complex needs of our filtering with greater efficiency, but to know, we need to be able to test it in-situ. However to do so would require to do the actual work necessary to facilitate the use of ElasticSearch for this cause. This epic is a plan to provide a path to a Minimum Viable Product using ElasticSearch for Vulnerability Search and Filtering. ## Potential Benefits * Expected Performance Gains - ElasticSearch is purpose built for high volume record filter and search. We have no stats, but there is expectation that it would outperform Postgres in this regard. * Improved Metrics - Our metrics currently push the limit of our ability to safely query in Postgres. ElasticSearch should theoretically be better at this as well. * Reduced Postgres load - Moving these demanding filter and search queries to ElasticSeach would reduce demand on Postgres, which may be beneficial to platform speed and stability. * Complex filtering possible - Filtering by values contained in an array while still paginating * Full Text Search - Postgres can support FTS to a limited degree, but ElasticSearch has very mature solutions in this regard. ## Identified Projects Elasticsearch has been mentioned in these projects. Likely because it will improve performance or extend our feature capability. 1. https://gitlab.com/groups/gitlab-org/-/epics/8089+ 1. https://gitlab.com/gitlab-org/gitlab/-/issues/477284+ 1. https://gitlab.com/gitlab-org/gitlab/-/issues/467675+ 1. https://gitlab.com/gitlab-org/gitlab/-/issues/514197+ 1. https://gitlab.com/gitlab-org/gitlab/-/issues/515333+ 1. Remove fuzzy counts https://gitlab.com/gitlab-org/gitlab/-/issues/438303 1. https://gitlab.com/gitlab-org/gitlab/-/issues/495640+ ## Vulnerability Report Filters 1. https://gitlab.com/groups/gitlab-org/-/epics/13340+ 2. <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/440182">OWASP 2021 filtering</a> 3. https://gitlab.com/groups/gitlab-org/-/epics/17251+ 4. https://gitlab.com/groups/gitlab-com/-/epics/2480+ 5. <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/520606/designs/Vulnerability_age_filter.png">Filter by vulnerability age</a> on the Vulnerability Report 6. https://gitlab.com/groups/gitlab-org/-/epics/13988+ ## MVC By %"18.0" aim is to have - [OWASP 2021 filtering](https://gitlab.com/gitlab-org/gitlab/-/issues/440182) - [Identifiers filtering for larger groups](https://gitlab.com/gitlab-org/gitlab/-/issues/514197) implemented for `GitLab.com` and `dedicated` customers on the Vulnerability report page at Project and Group levels using ES. #### MVC Plan: * a. %"17.9" & %"17.10" - ES infra configurations finalization, Schema finalization, continous updates and Backfill ingestion. * b. %"17.11" - Query building and GraphQL API's for the above 2 features. In this step we will also take help of insights team on GraphQL API's and Frontend. * c. %"18.0" - Full rollout of above features on gitlab.com and dedicated. #### High level architecture design: https://gitlab.com/gitlab-org/gitlab/-/issues/514697#note_2311629764 WIP: https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/vulnerabilities_filtering_with_es/content/handbook/engineering/architecture/design-documents/vulnerabilities_filtering_with_es/_index.md?ref_type=heads #### Project status: Latest status: https://gitlab.com/groups/gitlab-org/-/epics/13510#note_2444166069 | Milestone | Issues | Notes | |-----------|--------|-------| | %"17.9" | https://gitlab.com/gitlab-org/gitlab/-/issues/514697+ | Completed. Demo recording [here](https://docs.google.com/document/d/1QPGVXl4eqb0PZ_st5yjX02iTVrc3rnE8kL4Ofdoxxdc/edit?tab=t.0#bookmark=id.1xyl37eo1n9g). | | %"17.10" | \- https://gitlab.com/gitlab-org/gitlab/-/issues/515543+ <br>- https://gitlab.com/gitlab-org/gitlab/-/issues/515553+ <br>- https://gitlab.com/gitlab-org/gitlab/-/issues/515565+ | \- Closed <br> \- Closed <br> \- Closed | | %"17.11" | - https://gitlab.com/gitlab-org/gitlab/-/issues/526872+ <br> - https://gitlab.com/gitlab-org/gitlab/-/issues/533906+ <br> - https://gitlab.com/gitlab-org/gitlab/-/issues/515547+ <br> \- [Vulnerability Reads finder integration with ES mappers](https://gitlab.com/gitlab-org/gitlab/-/issues/524099) <br>- [GraphQL API for OWASP 2021](https://gitlab.com/gitlab-org/gitlab/-/issues/524127) <br>- [GraphQL API for identifiers filtering larger groups support](https://gitlab.com/gitlab-org/gitlab/-/issues/532695) | <br> \-Closed <br> \-Closed <br> \- Closed <br> \- Closed <br> \- Closed <br> \- Closed | | %18.0 | \- Frontend integration of [OWASP 2021 ](https://gitlab.com/gitlab-org/gitlab/-/issues/532703)& [Identifiers filtering](https://gitlab.com/gitlab-org/gitlab/-/issues/532716) <br>- FF enabling on gitlab.com | \- Closed <br> \- [Default enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/537673) from %"18.2" | ## Post MVC scope: - Self Managed instances feature enabling https://gitlab.com/gitlab-org/gitlab/-/issues/525484. Ongoing discussion in https://gitlab.com/groups/gitlab-org/-/epics/13510#note_2322497646 - Support count usecases from ES. https://gitlab.com/gitlab-org/gitlab/-/issues/480378 - Support Instance Security dashboard filtering with ES. https://gitlab.com/gitlab-org/gitlab/-/issues/537807 - Reduce ES sync delay. https://gitlab.com/gitlab-org/gitlab/-/issues/515543#note_2342251240 - Sync to ES without GitLab Rails application. ~~Ideas like using PG triggers and a seperate change table with its own sync service, external replication infra to capture change data. https://gitlab.com/gitlab-org/gitlab/-/issues/525131~~. Tracked now in its own epic https://gitlab.com/groups/gitlab-org/-/epics/17835+ ## Cost Implications There will be a cost implication to the storing of Vulnerability data in ElasticSearch. We are currently estimating a region of 95 - 125 million vulnerability records to be stored until our Vulnerability Retention policy is enacted in May 2025. This may result in the need to scale the ElasticSearch cluster, which will require an approval. @maddievn has expressed that this should be straightforward to manage if needed. ## Considerations **Note:** Below considerations has to be revised based on recent PG limitations. Read [thread](https://gitlab.com/groups/gitlab-org/-/epics/13510#note_2302208583) for details. * We cannot alienate or force the adoption of ElasticSearch for self-managed users in this implementation, which means any features we produce using ElasticSearch should ideally be available from Postgres only instances. * Ensuring feature parity between Postgres and ElasticSearch implementations means we need to able to toggle between DB provider at will. This could be facilitated by feature flag, but could also be facilitated by inheritable Namespace setting. This will also prevent a perpetual feature flag from existing. ## High Level Implementation Plan * ~database Implement Vulnerability Reference to store Vulnerability information in ElasticSearch * ~database Implement inheritable NamespaceSetting for Vulnerability Advanced Search (using ElasticSearch) * ~backend Implement ingestion of Vulnerability information into ElasticSearch during Vulnerability Ingestion process. * ~backend Implement updating of ElasticSearch references in Vulnerabiltiy information changes elsewhere in GitLab * ~backend Define ElasticSearch finder for vulnerability information with new capabilities (identifer, owasp, FTS), attach to the `vulnerabilities` GraphQL query respecting the namespace setting * ~frontend Push the Namespace Setting to the Frontend for the Vulnerability Report * ~frontend Implement Identifer, Owasp and FTS on the Vulnerability Report, as well as existing features, when the setting is enabled.