Add benchmark to CI test and document CI.

This invokes the benchmarks tool to do simple performance tests on the commit under test.

At the moment, the performance test can only be run on specific hardware, to keep performance tests as repeatable as possible. We only have one runner at the moment which can run them, so this is a single point of failure. However, the performance test isn't required to pass; it will only show as a warning if the performance test can't be run, or fails.

added 1 commit

• 684f2a3f - CI.tex: New file: Documentation for continuous integration

Compare with previous version

changed title from WIP: .gitlab-ci.yml: Add the performance test. to WIP: Add benchmark to CI test and document CI.

changed the description

added 12 commits

• 684f2a3f...be5a8638 - 10 commits from branch master
• ffb693b7 - .gitlab-ci.yml: Add the performance test.
• 2f90fc82 - CI.rst: New file: Documentation for continuous integration

Compare with previous version

resolved all discussions

added 1 commit

• 93acaa31 - Experiment to see if we can run user setup as a dependency

Compare with previous version

added 1 commit

• f9ebdd0b - Further experiment: Use YAML anchors to set up users

Compare with previous version

added 1 commit

• 28264029 - Remove user_setup dependency from CI YAML

Compare with previous version

added 2 commits

• e4be0906 - .gitlab-ci.yml: Add the performance test.
• d87a9223 - CI.rst: New file: Documentation for continuous integration

Compare with previous version

resolved all discussions

added 9 commits

• d87a9223...6e2a342d - 7 commits from branch master
• d28b594b - .gitlab-ci.yml: Add the performance test.
• 751a84eb - CI.rst: New file: Documentation for continuous integration

Compare with previous version

added 2 commits

• 7d781a79 - .gitlab-ci.yml: Add the performance test.
• 862d02fd - CI.rst: New file: Documentation for continuous integration

Compare with previous version

unmarked as a Work In Progress

added 39 commits

• 862d02fd...a4eb8bd2 - 37 commits from branch master
• 7925dd74 - .gitlab-ci.yml: Add the performance test.
• 5620cfd4 - CI.rst: New file: Documentation for continuous integration

Compare with previous version

added 39 commits

• 5620cfd4...e0a8b9e5 - 37 commits from branch master
• 08f64691 - .gitlab-ci.yml: Add the performance test.
• 7b71f503 - CI.rst: New file: Documentation for continuous integration

Compare with previous version

I made a comment on this but it was only on IRC, apparently my comments got lost there.

I don't see what we get out of adding this to BuildStream's CI.

• It's not really important to continuously run the benchmarks many times on the same machine continuously over time
  • Rather it's important to be able to run the benchmarks suite against various chosen versions of BuildStream on the same machine once, while nothing else is running and significantly skewing the result
  • If you want to run them for every merge request which lands in BuildStream master, it may be better to find another way, e.g. have the gitlab pipeline post the commit ID which landed in master over to another service which is doing benchmarks
• Presumably, this will slow down pipelines a lot, we are not going to block commits on running benchmarking suites
• In it's present form, all this gives me in the artifact on gitlab is a json file, this is quite meaningless to me.

closed