Skip to content

Draft: Add automatic snapshots and an HPA to Midgard

Robot Theory requested to merge rt-thor/node-launcher:midgard-hpa-merge into master

This PR combines Midgard and the timescaledb into a single pod. It takes periodic snapshots of the PersistentVolume that holds the database and uses this as the source for new copies of the pod. This makes the pod the "unit of scale", allowing you to create a new replica without syncing from 0.

The PR takes advantage of this by adding an HPA for the pod. It will automatically scale up when the CPU usage exceeds 50% (by default).

Some notes: 1) it depends on a golang utility ("snapper"). I am happy to add the source to this repo, but I don't know where it belongs. 2) Since the DB is part of the pod, the init script that paused Midgard start until the DB is ready is no longer useful. That means that, if the Midgard container starts before timescaledb, it will exit. The solution is to have Midgard retry the connection a few times before exiting. 3) Also, since Midgard and the db are in the same pod, it can use localhost. I left the existing DB service in place, but it is probably worth asking the question if this is needed.

Merge request reports