Decide what tool we should use to test CPU for that type of instance + implement it inside Nancy baseline check
Goal
When using AWS EC2 instances, we need understand as early as possible, if we deal with the instance which has worse characteristics as normal (as usually seen for such type of instances) due throttling or anything else.
Additionally, when conducting multiple series of experiments in parallel, we will need to adjust results and understanding of the instance's "performance shift" with respect to the baseline (how better or high worse it is compared to some "normal" values – values which we usually observe for this type of instances) will help to eliminate/ or smooth line abruptions in the final graph.
TODO / Implementation details
We started with sysbench, and have concerns about it (@Nastradamus @masterlee997 which ones exactly?)
We also considered pgbench, but it's higher level than needed – it will involve disk IO, but we want to have "pure" CPU/memory test.
perf
, memcpy
might be good candidates.
The test duration must not exceed 60 seconds. 10 seconds are less would be optimal.
Acceptance criteria
As a user of Nancy, during experiments, I can see that I'm dealing with an instance which is significantly (>10%) worse in performance than it should be. I see exactly, what is the reason -- disk IO (measured separately with fio), or CPU/memory.