Improve testing to anticipate possible bottlenecks on production DB/queue
Why?
Action item from FCL for incident 8619.
Definition of Done
- Create a proposal on how could we test and anticipate the impact of the introduced changes on DB/queue usage potentially could degrade performance.
- Introduce a proof of concept based on introduced and rollbacked MR: gitlab!114426 (merged)
Propose a data testing approach and explore potential options:
- Use real-world data: Use real-world data that accurately reflects the usage patterns of the system. This can be achieved by capturing and anonymizing production data for use in testing.
- Use synthetic data: Create synthetic data that simulates real-world usage patterns. This can be useful in identifying edge cases and unusual usage patterns.
- Use randomized data: Use randomized data to test the robustness of the system. This can be useful in identifying potential security vulnerabilities and unexpected behavior.
- Use database profiling tools: Use database profiling tools to measure the impact of changes on the database. These tools can provide detailed information on query performance and resource usage.