Speed up CI with incremental builds and tests (Bazel)
Problem to solve
For maximum correctness, CI/CD systems usually rebuild everything from scratch on every run. This is considered safer since artifacts from one pipeline won't negatively impact subsequent pipelines.
e.g. This is a classic problem in some legacy uses on Jenkins, where each pipeline run modifies the runner state. If a build fails, you never know for sure if it was because of something in the new code, or if something leftover from a previous run. Likewise, pipelines aren't completely repeatable since you never really know what the complete state was when it was run.
But re-running everything from scratch is very slow. We try to improve upon this by using caches and shared artifacts, but there's only so far that approach can go.
Bazel is a good example of tackling it a different way. Bazel speeds up builds by only rebuilding what is necessary. On the surface, that sounds a lot like just having a cache and doing an incremental build. But the main difference is that Bazel is really good at not only being "fast", but also "correct". It's much more reliable than traditional Makefiles or build scripts which are notorious for occasionally having to do a make clean
because they get into some inconsistent state they can't recover from.
Going further, since you know exactly what has changed, you can be more selective in doing incremental tests and have the confidence that skipped tests were truly unnecessary.
In an ideal world, after a minor change, the build and test should only take a few seconds. When it gets fast enough, it could even be triggered via an editor on every change before being committed to git at all.
Intended users
Further details
Proposal
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
- https://blog.bazel.build/2016/01/27/continuous-integration.html
- https://blog.donbowman.ca/2018/09/03/navigating-the-restrictions-of-ci-and-bazel/
- https://medium.com/@bamnet/using-bazel-on-gitlab-ci-fe59cad2db50
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.