Separate staging and release stages from building the package

Fixes #2071 (closed)

Purpose

In order to have a better control over our package release process, we need to remove some of the automation that is currently happening.

Current situation

We tag a release, packages are released to the public. We have no window of control while we are deploying to staging or production.

Intended behaviour

Build the packages ready for release, upload it to staging. Deploy to staging environment and production. After we are certain that we are ready to release, click on a manual task to release to public repositories.

Current state of the WIP MR

All packages (the screenshot only includes small subset) are first built. After that, they get uploaded to a staging (private) repository.

Docker image gets built and uploaded to staging registry.

Manual jobs, once started, would fetch the artifacts from staging and push to public repositories

Technical TODOs

• Code handling package push to package server needs to be covered in tests.
• The same code should then be prettified (if possible)
• Do the actual upload of the package, currently it is a puts
• Consider if 7 days of holding the artifacts is long enough For now, increase to 14 days.
• Implement the change for RPI builds too Do in a separate MR.
• Decide what we want to do with AWS builds Leave as it is.

Other TODOs

• Once merged, inform/implement a change in GitLab deployment infrastructure
• Write up documentation for Release Managers of the release where this MR will land
• Have a large beer to celebrate. I love dark beers so preferably a pint of that. I won't say no to wheat beer too.

FAQ

Q: Can't we just have one manual job that would upload all packages at the same time?

A: We can. But we shouldn't have that. If for some reason, one package fails to upload during the upload of all of them, we won't be able to upload the rest. This is because packagecloud api returns an error when pushing the package with the same name

Q: Aren't we doing duplicate work with uploading to two environments?

A: Yes we are.

Q: Are there better ways of handling this?

A: Probably. But I haven't heard any suggestion nor anyone tried implementing them.