Add a migrations specific queue for indexing

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Background

Corrective action from gitlab-com/gl-infra/production#19026 (closed)

There are Redis queues for used indexing data into Advanced search in a few different categories. The data indexed through the queues could be of any non-code type data.

1. initial - new data, imports, and migration framework for data migrations
2. incremental - updates existing data
3. embeddings - embedding data

Data migrations have the potential to back up the initial indexing queue and impact those indexing requests. This would be a poor experience for customers who are getting Advanced search for the first time because their data could be out of date

Proposal

Split data migrations into their own index to reduce impact for customers and initial indexing requests