Parallelize/improve VSA BG workers
This issue is a follow-up for Investigate slow reaggregation in VSA (#451275 - closed).
Todo: Reasses the aggregation situation after a few days/weeks after Investigate slow reaggregation in VSA (#451275 - closed) is closed.
Problems:
- The reaggregation worker has difficulties catching up, the data volume it needs to scan is too big.
- The incremental worker (ingesting new changes) has not around 30-40 minutes data lag.
Challenges:
- Since we insert data into PostgreSQL, the workers will use the primary DB. Parallelizing the workers could add pressure to the primary DB.
Things to consider:
- Move VSA to CH (in this case we would send data to ClickHouse directly, this means that we can use DB replicas). More parallelization.
- Improve the https://docs.gitlab.com/ee/development/database/efficient_in_operator_queries.html tooling to achieve faster looping.
Edited by Adam Hegyi