The new release roadmap

The besoin

Long time ago, we had few recalboxes pinging the archive.recalbox.com/..../recalbox.version file to know if an update were available.

We had some issues since recalbox became the most incredible retrogaming os of the world:

  • 2000 to 4000 recalboxes are connected to internet constantly, a stable release makes 4000 * 150mo (600 GO) to serve at the instant.
  • we fear the stable update as we have a process where unstable keeps going on for month before merging
  • we do not have process to test a stable release for only some recalboxes

So the need for recalbox update is the following:

  • no unstable branch. That means the ability to test dev branches really easily, but also to test the release on master branch (stable) for some recalboxes before letting it spread
  • many servers must serve files. This cannot be S3 or we are broke (seems like 700$/month is not a bad idea of what we should pay if we use it)
  • I want to be able to stop an update that is known to fail
  • I want updated recalboxes to tell me when an update has succeed or failed.

Steps to achieve this goal

Now that we can use docker images to deploy archive.recalbox.com, we need to deploy unstable versions in the form of containers to the rancher infra.

  • 1st step - the release of the 4.1 unstable on the recalbox infra (MR !117 (merged) !120 (merged) !121 (merged) !122 (merged) !123 (merged) and Issue #133 (closed))
    • today, the image recalbox/archive:4.0.2 serve the stable files (for noobs essentially). It stays here until we are ready.
    • the master release must be automatically released on a specific fqdn for each release. (like https://master-1234556-archive-prod.recalbox.com)
  • 2nd step - make possible branch update again (MR !124 (merged) and Issue #140 (closed))
    • we need to change the way recalbox check for upgrade when searching branch update: it uses the fqdn specified in recalbox.conf
  • 3rd step - use https://gitlab.com/recalbox/ops/recaleur to manage the new updates on master, with canary support (MR !129 (merged) and Issue #145 (closed))
    • when searching an update for master (stable) the recalbox must contact recaleur to see if an update is available.
  • 4th step - make noobs a part of the image (MR !131 (merged) !138 (merged) Issue #147 (closed) #156 (closed))
    • the image must contains noobs files
    • we must update noobs os.json by contacting them (this can be done in //)
    • noobs cannot use query params, we must provide a special path
    • the image must have old noobs compatibility (MR !144 (merged) Issue #164 (closed))
  • 5th step - archive.recalbox.com compatible docker image (MR !137 (merged) Issue #155 (closed))
    • the docker image for recalbox must be compatible with the old way to update.
  • 6th step - canary time based for archive.recalbox.com (MR !140 (merged) Issue #159 (closed))
    • as the first release of 4.1.0 will be on the old process, we need to limit the concurrent download when the update will be available. An other of recaleur will be responsible for that.
  • 7th step - release files
    • the release page of recalbox-os on github will contain raspberry pi zip for all rpi board, we need to create this file automatically
  • 8th step - make this project public (Issue #161 (closed))
  • 9th step - the release of the 4.1 stable on the recalbox infra
    • - one step : redirect archive.recalbox.com to master url.

And after :

  • logs
    • we need to log when upgrades are ok. That mean when someone starts an upgrade, we can notify a service, when the download is over, when the system reboots, and when it is back online.

How it works:

Forget the old process

When recalbox check for an upgrade it calls : https://recaleur-archive-prod.recalbox.com:9443/v1/upgrade?arch=BOARD&branch=BRANCH&boardversion=CURRENTVERSION&uuid=BOARDUUID&user=USER

  • branch will be stable for stable version (master), branch name for testing on merge request
  • arch the board type
  • boardversion will be used to know what is the current version installed. This is the build branch, date and id.
  • uuid is used to identify the board. It's not the /proc/cpuinfo, maybe a random uuid. With this data we can use canary release : only a percentage of uuid will receive the upgrade in the first hour of a release. As we put some logs in emulationstation for example, if Kibana reports all is ok, we can open the upgrade to everyone. This MUST be an automatic process at end.

The service at https://recaleur-archive-prod.recalbox.com:9443/v1/upgrade is aware of available upgrade. He knows that when you are on boardversion master-2017-08-24-123456 and you want to upgrade on master (you want a stable upgrade) he will check if the current version available is fresher, and return the url of the upgrade.

Upgrade url are on different subdomains of recalbox. This is the only way to make dns round robin when recalbox asks for a file, and prevent dns upgrade errors.

Here is a little story:

I am a recalbox on rpi3. I have stable in my recalbox.conf. My version is also master-2017-08-24-123456.

The last stable (master) build of recalbox has been done the 2017-08-24 and has been upgrade to https://master-2017-08-24-123456-archive.recalbox.com. I know that because i made an upgrade yesterday. I feel so fresh.

Still, I make a GET request to https://recaleur-archive-prod.recalbox.com:9443/v1/upgrade?arch=rpi3&branch=stable&boardversion=master-2017-08-24-123456&uuid=012456789&user=digitallumberjack@recalbox.com

The answer is 204: No Content as I already have the last version for the stable branch.

The day pass, here come the time of the apero. Many people play on me during the night. While we enjoying pixels explosions and stuff, a new build is done on master, deploying the new files to https://master-2017-08-26-891011-archive.recalbox.com.

In the morning, my people would make a last fight on SF3 3rd strike. I try to upgrade one more time.

I make a GET request to https://recaleur-archive-prod.recalbox.com:9443/v1/upgrade?arch=rpi3&branch=stable&boardversion=2017-08-24-123456&uuid=012456789&user=digitallumberjack@recalbox.com

The answer is 200 and the response has a body. OMG! A brand new version!

I get in the magnificent body a json with the new url to get when updating: https://master-2017-08-26-891011-archive.recalbox.com

So a begin to download the root.tar.gz and boot.tar.gz. What make the download so quick is that the domain master-2017-08-26-891011-archive.recalbox.com is pointing to many IP, and the dns round robin them. So I pick an ip among many, and do not slow down one server as i did before when there was only archive.recalbox.com.

I reboot, and when i'm back online and emulationstation is started and works, i send a little message to the log api of recalbox:

{ "board": "rpi3", "uuid": 012456789, "user": "digitallumberjack@recalbox.com", "boardversion": "master-2017-08-26-891011", "branch": "stable", "updateTimeInSeconds": 286, "status": "OK" }

I know this message will help the team being confident with upgrade process and make the update available for everyone!

Edited by digitalLumberjack