[WIP] CI is too slow

Created by: kfriedberger

WORK IN PROGRESS. This issue discusses the problem with the slow CI and possible solutions.

Use CCache for GCC and Clang

I tried to add ccache for the gcc and clang job. This does not bring any improvement, because we use the compiler only for the first step of the compilation (i.e., no linking), which does sadly not benefit from ccache. To test this on your own system, install ccache and use make CC='ccache gcc-5' for executing the CI-job.

Speedup the script compare.sh (see #879)

This script requires the most runtime and is mostly the reason for timeouts. It is written in BASH and is horribly inefficient in some parts. I will replace it with a faster Python-based implementation, see #879.

Split the benchmark set (at least for CI)

Another obvious possibility to improve the CI checks would be to generate more parallel CI jobs. As most categories are independent, this should be easy to do. The drawback is the maintainance cost, i.e., when renaming categories, directories, files. However, if the repository grows more and more, this will be the only scalable solution. With #821 this was partially done, i.e., the CI checks for one large set of new benchmarks are executed in a separate (parallel) CI job.

Exclude tasks from being checked again and again

For unchanged tasks we could store a hash and them on repeated checking. This would have some maintainance costs when changing makefile rules, compiler options, and with compiler updates.

Change to a faster CI hardware

In the future we might fully migrate to the GitlabCI, where we can include our own hardware. This might improve the runtime.