Test for DB regressions with a production-like DB and production-like traffic
Subtle changes in indices or query plans can have a dramatic effect on database performance but often can only be found if testing with production-like traffic against a production-like database setup as can be seen in this incident.
We should find a way to enable production-like regression tests.
This could be done like with gitlab.ai proposed here to some extend or by bringing up a benchmarking environment and replaying production like synthetic traffic like done here.