### Problem to solve Search Latency increases as GitLab.com scales. This will become worse as we add new content types to Advanced Search. Having everything in one index has long-term implications for the time to add new features. Anytime we add a new feature that adds new data, like a facet, we must process the entire index to backfill the documents with the new data. This process is estimated to take 20 days due to the need to throttle the indexing processes. Customers would experience an inconsistent experience and possibly a slower experience using Gitlab. Advanced Search is available to all Premium and ultimate customers on GitLab.com. These would be the most impacted by any outages that would occur if this change is not made. Update: 2022-06-04 : We need to prioritize this work, the [Average Shard sizes](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html) are now over 50GB, and we will see latency impacted during large indexing jobs or high traffic times. This will progressively become more frequent over the next 3 months. [This type of event happened in 2020, between Oct and December. ](https://gitlab.com/gitlab-org/gitlab/-/issues/292439) ### Intended users All users will take advantage of the new faster search. However, no specific persona is a direct focus. Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/ ### User experience goal There is no direct user experience goal. This epic is about making Search scalable but has the benefit of improving performance. ### Proposal De-normalize data. Instead of joining a parent project object, we embed the project authorization data onto each record associated with the project. We also separate each document type into different indices (e.g. issues, comments_on_issues, comments_on_merge_requests, merge_requests, ...) and alongside each record, there is stored a project_id,access_level pair. In this case, the access_level is not the project level access but the relevant access level for the feature so: issues index => access_level=issues_access_level comments_on_issues index => access_level=issues_access_level merge_requests index => access_level=merge_requests_access_level comments_on_merge_requests index => access_level=merge_requests_access_level ### Further details Since this will involve multiple changes, we will want to come up with an iterative approach. Some of the challenges will be using more than one index (as opposed to now where GitLab puts all record types in a single index). Second, we'll need to start storing the records differently. ### Documentation This will need to update the architecture documentation only. ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> ### Available Tier * Premium/Silver * Ultimate/Gold ### Feature Usage Metrics There is no specific feature usage for this maintenance epic. ### What does success look like, and how can we measure that? We can effectively scale up Advanced Search without impacts on latency. ### Is this a cross-stage feature? There is no cross-stage feature for this maintenance epic. ### What is the competitive advantage or differentiation for this feature? This maintenance epic is for scaling GitLab.com Advanced Search. ### Links / references <details><summary>Click to expand details for the original issue</summary> # Problem GitLab provides various authorization mechanisms for different resources (e.g., issues, merge_requests, comments...), and authorization is mostly derived from project membership or the visibility_level of features in a project. For example, a user may have access to 5 projects on GitLab.com and hence can search through all the issues on those projects, but they also should be able to search for all issues on public projects on GitLab.com. To perform our searches against Elasticsearch, we need to account for this authorization with data stored in Elasticsearch. Noting that we cannot filter results after they come out of Elasticsearch as pagination would be incorrect. It may have horrible worst-case performance characteristics if many early results are not visible to the user. One naive solution would be to store `project_id` for every document in Elasticsearch and then pass the full project_id to the search, but this would then mean potentially passing thousands of ids in the query for a big GitLab instance (like GitLab.com) that has many public projects. So the better approach is to only pass in the ids of all private projects the user has access to and also include results for public projects, which requires storing the project's visibility level in Elasticsearch. So now we need to store `project_id, visibility``.`.` But authorization turns out to be even more complex because public projects can have private issues or private merge requests, so we need to store the visibility level of all project features alongside the project. At this point, it becomes apparent that storing this in a normalized way with a parent project is beneficial. And hence we end up using joins to parent types in Elasticsearch. # What we do today Today we store all our data in a single index joined to a parent type of project. This join approach (i.e., normalized data) has the advantage that you only store information about a project's authorization rules once and hence lower storage and overhead when updating. It's worth noting that, in general, Elasticsearch encourages people to de-normalize their data to improve performance. Generally, the normalized data with joins has some serious limitations, including forcing projects to share the same shard, which can easily lead to shard imbalance. It also forces us to store everything in a single index, which means scaling and monitoring separate parts of the index are difficult, and searches may be slower because they're searching across more data. Also, we need to store document types with every document and have a shared schema for all documents that are a superset of all the different fields needed by all documents being stored. # Where we want to get to Since there are 2 problems with denormalized data that I believe will both be easy to overcome, I believe we should move to denormalized data: 1. Denormalized data takes more storage => Separating indexes will mean storing only 2 integers alongside each document which is a very tiny amount of data and should not be noticeable 1. Updating all such records is expensive and tricky => Using the [Update by Query API](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/docs-update-by-query.html) should make it simple to find all matching records for the project and update them and performance should not be much of a concern because project visibility/access level changes should be very infrequent and especially so for very large projects with many associated records. For example, it's unlikely we'll make issues private on `GitLab-org/GitLab any time soon, but even if we did it occasionally, it should not take down Elasticsearch. De-normalize data. Basically instead of joining to a parent project object we embed the project authorization data onto each record associated with the project. We also separate each document type as well into different indices (eg. issues, comments_on_issues, comments_on_merge_requests, merge_requests, ...) and alongside each record there is stored a `project_id,access_level` pair. In this case the `access_level` is not the project level access but the relevant access level for the feature so: - `issues` index => `access_level=issues_access_level` - `comments_on_issues` index => `access_level=issues_access_level` - `merge_requests` index => `access_level=merge_requests_access_level` - `comments_on_merge_requests` index => `access_level=merge_requests_access_level` It's also worth noting that Elasticsearch does support querying multiple indices, so should we want to search all comments at the same time, it's as simple as searching `comments_on_issues,comments_on_merge_requests``,``,`,`` and the upside of the fact that `access_level` is named the same in all means that we don't need to account for both kinds of access levels in the query. # How to get there? Since this will involve multiple changes, we will want to develop an iterative approach here. Some challenges will be that we need to switch to using more than one index (as opposed to now, where GitLab puts all record types in a single index). Second, we'll need to start storing the records differently. From what I can tell, the simplest way to approach this iteratively is to tackle this one document type at a time. Given our work on the Admin pages, which are designed to support multiple indices, we may need to adapt that a little as this new architecture will mean that there are always many indices, and migration won't just be from one index to the other. </details>