@@ -53,3 +53,11 @@ When conducting experiments manually (with or without Nancy) it is important to
...
@@ -53,3 +53,11 @@ When conducting experiments manually (with or without Nancy) it is important to
# commands ...
# commands ...
) 2>&1 | ts | tee-a logfile.log
) 2>&1 | ts | tee-a logfile.log
```
```
# Main principles of collecting and analyzing results
1.**Determine bottlenecks.** Were we CPU- or IO-bound? Did you have significant steal time?
1. Collect **statistically meaningful data**:
* do not change environment when comparing: if baseline run is to be compared with some changed (with some "delta") ones, always do it on a single machine. Example: we need to compare behavior under two conditions: `max_wal_size='1GB'` and `max_wal_size='64GB'`. This should be done on a single machine, in the same environment.
* do enough repetitions: conduct the whole experiment at least 4 times, in different machines, of the same type. If we want to draw a "global" conclusions, we need to try at least a few *different* machines (of the same type first!; and only then, if needed, on various types).