Automation for the update of JMeter test plans from Production Workload sampling

As discussed at https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16376#note_1204631752 the update of the JMeter test plans manually would be a massive work that needs to be performed periodically, therefore we decided to automate the process as much as possible.

On a top level the automation should be done in SQL, Bash and/or Ruby whenever possible.

The major steps of the process are:

  1. Sample queries metrics from pg_stat_statements (aka PGSS)
  2. Sample a production workload (including arguments) from pg_stat_activity (aka PGSA)
  3. Load the sample data into a database in the benchmarking environment
  4. Filter PGSS to get the a top percentage of queries based on their total_time or calls
  5. Parse PGSS and PGSA samples to match them through their fingerprints (hash of the normalized statement)
  6. Parse PGSA samples to gather arguments for a given query
  7. Generate JMeter plans (in XML) and the respective queries argument files (in CSV)

For the parsers we are using pg_query Ruby lib

Edited by Rafael Henchen