Hard-coded SV-COMP iteration year in benchmark definitions
We currently have run definitions inside the benchmark definition files being named with the SV-COMP iteration year as prefix, e.g.:
<rundefinition name="SV-COMP23_unreach-call">
This needs to be updated every year, which is tedious and can cause merge conflicts (e.g. !278 (comment 1137376188)).
It would be better to make the files agnostic to the SV-COMP year (after all, nobody is prevented from runnning these benchdefs with a different version of the sv-benchmarks repository, so this is even dangerous!) and let the person who executes the benchmarks instead add this meta-information by using benchexec
's option to give a custom name (prefix) to the executed run collection
-n NAME, --name NAME Set name of benchmark execution to NAME
This might cause incompatibilities with existing (third-party?) scripts that rely on the exact naming convention in the run definition names of SV-COMP though. Ideally benchexec
would provide a way to prefix the run definition names directly, such that this can be done as a pure refactoring.