We are in the process of rolling out Elasticsearch based Advance Global Search on GitLab.com. This Epic is created to be a central place where we can see the rollout progress, related issues, and other relevant information. ## Timeline Finished: \~ 2020-07-15 * [X] 2020-06-09 : => 31% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2233 * [X] 2020-06-12 : baseline comparison of scale again => 32% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2234 * [X] 2020-06-15 : experiment to scale up sidekiq => 33% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2209 Now that we've concluded it is safe to run at 2x scale from the above experiment * [x] 2020-06-15 : => 38% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2275 * [x] 2020-06-16 : experiment to scale up sidekiq again => 39% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2276 Now that we've concluded it is safe to run at 2x scale (again) from above experiment * [x] 2020-06-17 : => 44% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2279 * [x] 2020-06-16 : => 60% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2280 * [x] 2020-06-17 : => 100% bronze https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2287 Assuming we've merged https://gitlab.com/gitlab-org/gitlab/-/issues/205178 https://gitlab.com/gitlab-org/gitlab/-/issues/207494 (scheduled for %"13.1" but they are quite complicated so may end up being merged in %"13.2") * [x] 2020-06-22 : => 50% silver https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2307 - [x] Assuming we figure out what went wrong in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2318 - [x] Complete re-indexing in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2335 to mitigate https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2318 * [x] 2020-07-06 : => 100% silver https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2371 * [x] 2020-07-08: => 50% gold https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2372 * [x] 2020-07-09: => 100% gold https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2373 Assuming by now we've merged https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35172 then all we need is to top them back up to 100% again to make sure ever previously created group is in the roll-out * [x] 2020-07-10 : => 100% gold, 100% bronze, 100% silver ## Rollout Plan We started the rollout by [dogfooding the feature within GitLab](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1608). Then, we are gradually enabling the feature to paid groups on GitLab.com. During the progress, we have been able to identify issues and have fixed those issues. We keep [iterating](https://about.gitlab.com/handbook/values/#iteration) on this progress and will eventually get the feature to all paid groups on GitLab.com. Based on our data at the beginning of this rollout, we have around [10k paid groups on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/118571#note_262151750). From the experience in the past few iterations, We will be adding the feature to 400 groups each week and plan to finish the whole process in September. ### Optimization As mentioned above, we have identified a few areas where we have been doing improvement now, which will help us speed up the rollout eventually. #### Scale the indexing by utilizing ES Bulk import API and Redis ZSET for indexing job deduplication https://gitlab.com/gitlab-org/gitlab/-/issues/34086 ### Reduce the Elasticsearch cluster storage size By implementing the changes listed in the following issues, we have seen [significant storage space reduction](https://gitlab.com/gitlab-org/gitlab/-/issues/27918#note_353043618). * https://gitlab.com/gitlab-org/gitlab/issues/28085 * https://gitlab.com/gitlab-org/gitlab/issues/27918 * https://gitlab.com/gitlab-org/gitlab/issues/27789 ### Separate initial indexing for new groups from incremental updates for existing groups One issue the team has seen is the long outstanding initial indexing queue could block the incremental updates from existing groups. It prevents us from adding many more groups at once. By solving the following issues, we hope we can speed up the rollout by adding more groups in each batch. * https://gitlab.com/gitlab-org/gitlab/-/issues/34086 * https://gitlab.com/gitlab-org/gitlab/-/issues/205178 * https://gitlab.com/gitlab-org/gitlab/-/issues/207494 ### Add more Sidekiq worker We are also investigating adding more processing power to speed up the indexing. However, it could have implications on overall GitLab.com performance. We have been very cautious about this. The team has been collaborating with other teams to tune the parallelism up. * https://gitlab.com/gitlab-org/gitlab/-/issues/214280 * https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/377 ## Rollout Dashboard We have set up a dashboard where we can see the total number of enabled groups and the history of the rollout process, https://app.periscopedata.com/app/gitlab/596072/Enablement:Search-Metrics ## Overall storage stats | | namespaces | projects | repository size | issues | merge requests | comments | | ----- | ----- | ----- | ----- | ----- | ----- | ---- | | Currently in index | 6444 | 261350 | 9.58 TB | 2.95 M | 6.1 M | -1 | | All of Silver | 2215 | 143890 | 3.26 TB | 1.17 M | 3.5 M | -1 | | All of Gold | 1054 | 66121 | 1.58 TB | 443 K | 720 K | -1 |