Benchmarks/watch_regressions: add a file with a selection of comparisons
Context
When receiving feedback after a run of the benchmarks, we get a report, all.csv
, with the value of all parameters on all runs.
We also get an alert file with the list of parameters with a regression. We can then check the evolution of the value of these specific parameters in all.csv
, for analysis.
This request adds a report, selected.csv
, that filters all.csv
with only the lines with a parameter with a regression. That removes the small burden of searching in all.csv
and speeds up analyses a bit.
Manually testing the MR
After downloading and extracting this set of data, and applying this patch that removes Slack messaging and AWS interactions, applies the fixes from !7824 (merged), and sets variables according to my configuration (adjusting OCTEZ_DIR
and INPUT_CSV_DIR
should be enough), then running watch_regressions
leads to this report, where we can see a reduced selection of parameters.
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rst
for the protocol and the environment,CHANGES.rst
at the root of the repository for everything else). -
Select suitable reviewers using the Reviewers
field below. -
Select as Assignee
the next person who should take action on that MR