... | ... | @@ -59,10 +59,16 @@ When conducting experiments manually (with or without Nancy) it is important to |
|
|
1. **Determine bottlenecks.** Were we CPU- or IO-bound? Have we noticed significant steal time?
|
|
|
|
|
|
1. Collect **statistically meaningful data**:
|
|
|
1. do not change environment when comparing: if baseline run is to be compared with some changed (with some "delta") ones, always do it **on the same machine**. Example: we need to compare behavior under two conditions: `max_wal_size='1GB'` and `max_wal_size='64GB'`. This should be done on a single machine, in the same environment.
|
|
|
1. do not change the environment when comparing: if baseline run is to be compared with some changed (with some "delta") ones, always do it **on the same machine**. Example: we need to compare behavior under two conditions: `max_wal_size='1GB'` and `max_wal_size='64GB'`. This should be done on a single machine, in the same environment.
|
|
|
1. do enough repetitions: conduct the whole experiment **at least 4 times**, in different instances of the same type. If we want to draw a "global" conclusions, we need to try at least a few *different* machines (of the same type first!; and only then, if needed, on various types).
|
|
|
|
|
|
In short:
|
|
|
- know your bottlenecks;
|
|
|
- compare on the same machine;
|
|
|
- do it 4+ times. |
|
|
\ No newline at end of file |
|
|
- do it 4+ times.
|
|
|
|
|
|
# Userful Links and Info
|
|
|
|
|
|
- pgbench's `-s` (scale factor): the resulting DB size will be `scale factor * ~16MiB` (example: for `-s100`, DB size will be ~1600 MiB)
|
|
|
- types of EC2 instances: https://ec2instances.info
|
|
|
- types of RDS instances: https://ec2instances.info/rds/ or http://rdsinstances.info/ |
|
|
\ No newline at end of file |