Skip to content

Draft: First pass at using a tarball cache

Sean Gregory requested to merge sgregory2/omnibus-gitlab:sg/use_tarball_caching into master

What does this MR do?

This Draft MR aims to make omnibus-gitlab isolate software build artifacts.

It uses https://gitlab.com/sgregory2/omnibus/-/tree/sg/isolated-build-cache (branched from omnibus!45 (closed)) to

  • add upstreams to a software's shasum so that changes to an upstream only rebuilds related components.
  • change Omnibus::Config to allow building with a tarball_cache
  • add methods to Omnibus::Software to route software builds to an isolated cache directory

The changes to omnibus-gitlab

  • add cache callbacks for strategies to fetch and restore build artifacts from the cache.
  • update the build definitions to install software to an isolated directory for caching
  • adds a cache key to .gitlab-ci.yml

This MR aims to speed up the build, primarily in cases in which a dependency has changed further up in the graph.

Concerns

While nice to prove out and be 1:1 with what is distributed, I believe that the modifications I’m making are actually more risky than some existing efforts to reduce build times in MRs, but contribute less to reducing build times.

For instance, this strategy employs DESTDIR to create staged build with the same prefix.

There are 6 or so software builds that do not honor DESTDIR, such as OpenSSL. In such cases the prefix is changed to build directly to the stage directory, and symlinks are reconstructed when restoring the cache.

I do want to follow up with a concurrency mechanism to omnibus, which will also be intrusive - (like this MR), but with much more reduction of build times.

I also have concerns with the testability and complexity in the omnibus fork and chef/omnibus in general. Would love to hear thoughts.

Related issues

Closes #7461

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for the GitLab Chart opened
Edited by Sean Gregory

Merge request reports