Improve build caching to speed up package compile times
Problem
Building an Omnibus GitLab package can still take 30min+, even after the optimizations we have done in the past. This has a number of drawbacks:
- It takes a long time for a developer to build a package off their branch, reducing the frequency this is done and impacting our ability to more rapidly and easily iterate
- We estimated each package to cost around 1-2 euros to create, largely due to the compute costs. If we can improve our caching, we can reduce the compute costs required to create one, and save money. This will also allow us to build packages automatically, in some pipelines potentially.
- This has also impacted our ability to quickly iterate in some release scenarios, having to wait for long pipelines to complete before being able to proceed.
Overview
We've worked on this multiple times in the past, we have multiple MR's and issues in this repository trying to and improving situation with our build caches, e.g. !1356 (closed) .
Current state:
- We cache assets,gems,omnibus git cache
- Each distro has its own cache
- Restore it from an S3 bucket in every build
- Save the new cache as cache that runner then pushes back to S3
This helps us speed up things quite a lot but it does suffer from some issues:
- The cache keeps growing with each build
- omnibus git cache is not smart, it gets invalidated easily
- Once invalidated, builds still take time as each library is rebuild
- We use very powerful machines to compensate this fact
- Each branch uses the same cache which means that multiple different releases invalidate cache easily
We very briefly looked into distcc
and ccache
and that gave us some great results but we never had time to see if we can implement it in full.
With the cost of each build rising and more urgent need to enable package/QA tasks on every MR, we need to come back to this soon.
Proposal
We have determined that the webpack compile time is currently the bottleneck in the process, and will investigate solutions with the goal of implementation in this release to reduce the compile time.