Randomly delay SQL queries and Git operations
Most developers will use a small database on their local machine. As a result the code they write is most likely based on the assumption that queries finish quickly and that there's no networking/IO overhead. This results in developers writing inefficient migrations and inefficient application code.
The best way to deal with this is to make developers feel the pain of their code running in the real world. To do so we should randomly delay both SQL queries and Git operations. The idea behind this is that the more operations one adds, the more they're going to feel the pain.
One incredibly crucial aspect of this is that this is enabled by default in the "development" mode (disabled in other modes) and that you can not opt out. The latter is hugely important as it forces developers to deal with the problem instead of just disabling it and pretending all is well. Of course one could just remove the code and commit their changes, but this at least requires a bit more effort than flipping some kind of switch.
The way I envision this system is that there's a default chance/delay, with the option of adjusting this for certain parts. For example, we may want to increase the delay for queries to the projects
table to simulate a larger table. Every delay would consist out of a time (in milliseconds) and a percentage indicating the likelihood of the delay being applied. Let's say the default delay is 10 milliseconds and has a 40% chance of being applied. The idea here is to prevent developers from getting used to certain code always being slow. This in turn is to simulate network: a network usually isn't always slow in the same way, instead this will vary depending on load and other factors.
For an MVP it's probably best to just start with a delay of N milliseconds that always takes place, then we can apply the chance factor.
This should be implemented in GitLab itself (so in Ruby) and preferably at the lowest level possible. For SQL queries I'd prefer to have a greater delay for writes (INSERT/UPDATE/DELETE) to simulate the need for updating indexes.
cc @pcarranza