Speed up CI with incremental builds and tests (Bazel)
Problem to solve
For maximum correctness, CI/CD systems usually rebuild everything from scratch on every run. This is considered safer since artifacts from one pipeline won't negatively impact subsequent pipelines.
e.g. This is a classic problem in some legacy uses on Jenkins, where each pipeline run modifies the runner state. If a build fails, you never know for sure if it was because of something in the new code, or if something leftover from a previous run. Likewise, pipelines aren't completely repeatable since you never really know what the complete state was when it was run.
But re-running everything from scratch is very slow. We try to improve upon this by using caches and shared artifacts, but there's only so far that approach can go.
Bazel is a good example of tackling it a different way. Bazel speeds up builds by only rebuilding what is necessary. On the surface, that sounds a lot like just having a cache and doing an incremental build. But the main difference is that Bazel is really good at not only being "fast", but also "correct". It's much more reliable than traditional Makefiles or build scripts which are notorious for occasionally having to do a
make clean because they get into some inconsistent state they can't recover from.
Going further, since you know exactly what has changed, you can be more selective in doing incremental tests and have the confidence that skipped tests were truly unnecessary.
In an ideal world, after a minor change, the build and test should only take a few seconds. When it gets fast enough, it could even be triggered via an editor on every change before being committed to git at all.