## Problem Today we have split out Notes, Merge Requests, and Issues into their own index while the remaining data types share the main index. Our major motivating factor for moving to indices for each data type is performance. The data types that have already been moved have seen improvements in response times. We think that total response time for `wiki_blob` searches can be reduced if those `wiki_blob` documents are in a dedicated index rather than the main index with all the other data types. ## Solution Move all `wiki_blob` documents to a dedicated index. This will have several challenges that we need to overcome and may require several MRs to accomplish. ## Proposal (issues will be opened for each) - Denormalise the `wiki_blob` permission data. This may be done over a few MRs which should be completed in order: 1. Change project and group-wide searches to use the existing `rid` field already on the `wiki_blob` documents in place of the parent project join: https://gitlab.com/gitlab-org/gitlab/-/issues/344471 1. Update the Elasticsearch indexer to send `visibility_level` and `repository_visibility_level` fields when indexing wiki_blobs. Create a new tag once merged.: https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/issues/88 1. Send `visibility_level` and `repository_visibility_level` data to the Elasticsearch indexer when `wiki_blob` documents are indexed. Trigger updates to all associated `wiki_blob` documents whenever the project permissions change. Update `GITLAB_ELASTICSEARCH_INDEXER` to use the new tag created in the prior step. This work must be done in the same MR to avoid project permissions getting out of sync: https://gitlab.com/gitlab-org/gitlab/-/issues/344580 1. Create an Advanced Search migration to update all the existing `wiki_blob` documents with the project permissions fields: https://gitlab.com/gitlab-org/gitlab/-/issues/344581 1. Change global searches to use the new denormalised fields rather than the join to parent: https://gitlab.com/gitlab-org/gitlab/-/issues/344583 - Move all of the `wiki_blob` documents to a new index 1. Update the Elasticsearch indexer write `wiki_blob` documents to the new index. Create a new tag once merged. https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/issues/89 1. Add an Advanced Search migration that pauses indexing, creates the new index (with `wiki_blob` mappings), moves the `wiki_blob` documents over to the new index, update the `GITLAB_ELASTICSEARCH_INDEXER` to use the tag created in the previous step ( https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/issues/89), and ensure reads and writes for `wiki_blob` are now coming from the new index (This may be done over a few MRs, but they **must be merged together**) https://gitlab.com/gitlab-org/gitlab/-/issues/344590 1. Add an Advanced Search migration that deletes the `wiki_blob` documents from the main index: https://gitlab.com/gitlab-org/gitlab/-/issues/344591