Create a package/image for easy testing of changes in CE/EE
Problem
Most developers do not know how to build the package to test their changes. First time they see their changes is when it hits the staging environment but in majority of the cases they see them when production is deployed.
Building your own package is documented, but this still requires some effort to get done.
Goal
Create a package or a Docker image that a developer can easily fetch to check out their change.
Challenges
- We build packages only on dev.gitlab.org, description of the infrastructure
- It takes around an hour to build the packages and an image
- Package needs to be accessible to public but also not be stored for a long time as it is temporary
- We only need one type of package, we don't need to make packages for all platforms
- Package needs to be built with all dependent components with the correct version to get a valid working package
Getting there
I propose we expand our build system on GitLab.com to build only Ubuntu 16.04 package and a Docker image and trigger a build from CE/EE in omnibus-gitlab.
Work necessary in omnibus-gitlab:
- Create new infrastructure for this purpose only. We can use kubernetes executor and persist cache to cut build times
- Make
custom_sources.yml
more flexible or better yet, ignore it if we provide ENV variables. This would allow us to receive the correct versions from the build trigger. - Add 2 more items in the gitlab-ci.yml that would only run on gitlab.com AND only on trigger. We have TRIGGERED variable provided by our CI so we can leverage that.
- Build Ubuntu 16.04 only, and build Docker image
- Add expiry of 1-2 days for the artifact
- Build docker image from the package
- Push to the registry of a separate repository to allow for quick cleanup. I propose doing this in CE and EE repos because they don't have registry enabled.
Work necessary in GitLab CE/EE
- Add 1 job in gitlab-ci.yml, first job and allowed to fail to make sure we don't block the rest of the tests.
- This job needs to:
- Read all VERSION files in the root directory and convert the values to ENV variables
- Pass those values and also the name of the branch (or SHA of the commit) to the trigger
- Trigger can be a simple curl call that will trigger omnibus-gitlab build on .com
FAQ
- Package build time can be an hour, do we want to wait for this?
- Well yes! If we put this trigger as the first job of the pipeline, the job would be done in parallel with other specs. All other specs take around an hour to complete so by the time they complete, package is ready.
- We would need to decide when this trigger happens (every push, new branch) otherwise our pipelines would be clogged constantly.
- If we have a dedicated runner manager for this process only in omnibus-gitlab, we should be fine with even building on every push. We can refine this when needed.
- Won't this create hundreds of MBs of artifacts?
- Yes it will but if we set reasonable expiry time, we shouldn't care.
- That won't work for Docker, we don't have an automatic cleanup.
- We don't but we can push to a repository that doesn't have images currently and then use the bulk deletion from the interface to clean up from time to time. We might even have a way of doing this automatically on schedule through API. If currently not possible, it should be at some point.
- Won't there be problems when omnibus doesn't have support yet for whatever I built?
- Yes, the package/image build will probably fail OR it will create an image that doesn't work. But then you can create a MR in omnibus to fix it :)
Opinions or better yet, better ideas @gitlab-build-team @regisF @stanhu @rymai?
Update
TODO
-
Allow alternative sources switch with one flag: #2165 (closed) -
Read versions of components from env variables if they are provided: #2249 (closed) -
Add to CI package build a section that will build a 16.04 package AND a Docker image: #2250 (closed) -
Use the concept made by Sean #2234 (comment 27729694) and add to CE and EE repos