Migrate blobs to dedicated Elasticsearch index

Proposal

For the many reasons described in #3217 (closed) and &2054 (closed) we want to move blobs to a dedicated Elasticsearch index.

Benefits

As described in &2054 (closed) and #3217 (closed) there are many performance gains we'll get from moving blobs to a separate index.

One major reason to start with blobs is that this is by far the largest part of our current index. So if we move that to a separate index then everything left in the current index will be considerably smaller and searches for those documents will be much faster.

Additionally once we have blobs in a separate index and a good migration strategy implemented #234046 (closed) we'll then be able to start implementing more advanced code search features like #223150 or #4175 (closed) .

Technical steps

  1. Write a migration to move the blobs data to a new index. Some ideas for this in #234046 (closed)
  2. Search will need to search existing index as well as the new index during the migration process
  3. Later ship another migration that deletes blobs from the original index

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by 🤖 GitLab Bot 🤖