Geo Docs Feedback: Zero Downtime Updates with Geo Deployments
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
This is a list of feedback for the documentation for updating a Geo deployment with Zero Downtime. As it stands the Geo sections of the docs are heavily reliant on the people knowing the non geo process as well.
Most of the issue stem from us having our own version of the process for Geo which lacks the detail of the original non Geo version. The update process for Geo is nearly the same as non Geo except for an extra migration command we run after updating both sites. Personally a preferred version of these docs would just be to use the non Geo version and have a couple of notes at points where Geo differs as this should only be in a couple of places.
Can machines be updated in parallel
In the non Geo version we call out that for sidekiq/puma commands should be run sequentially
Complete the following steps on the other Puma/Unicorn/Sidekiq nodes, one after another. Always ensure at least one of such nodes is up and running, and connected to the load balancer before proceeding to the next node.We don't mention this in the Geo docs. When trying to run this process with automation it would help a lot if we called out if every component should be updated sequentially or can some be done in parallel. For most nodes we just state On all other nodes excluding the primary “deploy node”.
Gitaly
For primary and secondary sites we call out what to do On primary Gitaly only nodes, this is only valid for non Gitaly cluster setups. at this point we should be linking to the Gitaly Cluster section as well as non cluster version.
Load Balancer
When restarting puma with sudo gitlab-ctl hup puma this will cause downtime, we need to call out that each puma node should be removed from the load balancer before performing the update, once the restart is complete the node can be re added and the next node removed.
/cc @nhxnguyen @fzimmer