--random-seed is insufficient to be able to debug or benchmark deterministically
As noted here: !1317 (comment 718905893)
Getting remotely deterministic benchmarks requires a lot more invasive changes than simply setting the RNG's seed, because different parts of the engine update at different rates, mechanics update with real delta time and cause different amounts of real time to pass between different amounts of RNG calls, etc.
Even after forcing the engine to use a fixed mechanics timestep equal to the physics timestep, and running the game at the same framerate as the mechanics timestep, things still aren't deterministic, because lots of completely unrelated things all use the same global RNG, and end up pulling from it in a non-deterministic order, or non-deterministic numbers of times, causing RNG nondeterminism when the RNG is deterministically seeded.
To summarize, the simplest way for me to get a deterministic benchmark was like this:
- completely dummy out all of the RNG functions, returning their average/mean value every time
- force the mainloop to operate on fixed timesteps instead of delta time, completely ignoring the actual framerate
- recast was turned off
- I don't remember whether async physics were turned on or off, but if they were on, they were set to use at most 1 single thread
For a more proper solution, every individual "type" of RNG usage should have its own internal state (UI, rendering, engine-side mechanics, scripting, etc).
There's also the issue that, for benchmarking, things that use the current time to affect game state should be given a deterministic version of the current time rather than the current "wall time", for the same reason being able to set the random seed is desirable, but I didn't run into anything during my rudimentary physics tests that necessitated this.