Automation for the update of JMeter test plans from Production Workload sampling
As discussed at https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16376#note_1204631752 the update of the JMeter test plans manually would be a massive work that needs to be performed periodically, therefore we decided to automate the process as much as possible.
On a top level the automation should be done in SQL, Bash and/or Ruby whenever possible.
The major steps of the process are:
-
Sample queries metrics from pg_stat_statements (aka PGSS) -
Sample a production workload (including arguments) from pg_stat_activity (aka PGSA) -
Load the sample data into a database in the benchmarking environment -
Filter PGSS to get the a top percentage of queries based on their total_time
orcalls
-
Parse PGSS and PGSA samples to match them through their fingerprints (hash of the normalized statement) -
Parse PGSA samples to gather arguments for a given query -
Generate JMeter plans (in XML) and the respective queries argument files (in CSV)
For the parsers we are using pg_query Ruby lib
Edited by Rafael Henchen