Spike: Idea to Parallelize Omnibus Building and Reduce Omnibus Upgrade Diffs

Spike final outcome

The work and discoveries from this spike produced the plan that will be executed in Distributed Asynchronous Builds in Omnibus GitLab (gitlab-org&12054 - closed) • Unassigned

Assumptions Describing the Problem Set

• Omnibus builds are becoming too slow for efficient testing
• Customers have to download the entire Omnibus package every time for upgrades which, though good for AirGap environments is not good for other environments and costs us in bandwidth
• Pi Builds are becoming problematic due to build times

Proposed Solution

• Break Omnibus builds up by dependency and parallelize the builds
• Ship Omnibus both as a single package (easy for airgaps) and as a meta-package that knows the dependencies and installs them all

Iterative Steps

The first step would be to take a single build and as a proof of concept show that we can do a build using our simple build method for testing software dependencies and deliver that as an artifact package to the next pipeline.

If this works for us, then we can expand this approach and parallelize all builds with our existing Omnibus software definitions. This could then be transitioned in a final step to create the meta-package.

cc @gitlab-org/distribution @ljlane @mendeni