... | ... | @@ -60,7 +60,7 @@ When conducting experiments manually (with or without Nancy) it is important to |
|
|
|
|
|
1. Collect **statistically meaningful data**:
|
|
|
1. do not change environment when comparing: if baseline run is to be compared with some changed (with some "delta") ones, always do it **on the same machine**. Example: we need to compare behavior under two conditions: `max_wal_size='1GB'` and `max_wal_size='64GB'`. This should be done on a single machine, in the same environment.
|
|
|
1. do enough repetitions: conduct the whole experiment **at least 4 times**, in different machines, of the same type. If we want to draw a "global" conclusions, we need to try at least a few *different* machines (of the same type first!; and only then, if needed, on various types).
|
|
|
1. do enough repetitions: conduct the whole experiment **at least 4 times**, in different instances of the same type. If we want to draw a "global" conclusions, we need to try at least a few *different* machines (of the same type first!; and only then, if needed, on various types).
|
|
|
|
|
|
In short:
|
|
|
- know your bottlenecks;
|
... | ... | |