... | ... | @@ -52,4 +52,12 @@ When conducting experiments manually (with or without Nancy) it is important to |
|
|
echo "step 2"
|
|
|
# commands ...
|
|
|
) 2>&1 | ts | tee -a logfile.log
|
|
|
``` |
|
|
\ No newline at end of file |
|
|
```
|
|
|
|
|
|
# Main principles of collecting and analyzing results
|
|
|
|
|
|
1. **Determine bottlenecks.** Were we CPU- or IO-bound? Did you have significant steal time?
|
|
|
|
|
|
1. Collect **statistically meaningful data**:
|
|
|
* do not change environment when comparing: if baseline run is to be compared with some changed (with some "delta") ones, always do it on a single machine. Example: we need to compare behavior under two conditions: `max_wal_size='1GB'` and `max_wal_size='64GB'`. This should be done on a single machine, in the same environment.
|
|
|
* do enough repetitions: conduct the whole experiment at least 4 times, in different machines, of the same type. If we want to draw a "global" conclusions, we need to try at least a few *different* machines (of the same type first!; and only then, if needed, on various types). |
|
|
\ No newline at end of file |