Enable GKE Auto Upgrades for Master and or Nodes
This alert popping up in ops https://gitlab.slack.com/archives/C12RCNXK5/p1597717867024700 made me curious so I did some digging to discover that all our Kubernetes clusters (ops, gprd, gstg, pre) which should be running version 1.16.10-gke.8
(due to #889 (closed)) are now actually running 1.16.11-gke.5
. They auto upgraded themselves sometime last week (e.g. for gprd)
The release notes here seem to indicate we were running a specific version targeted for upgrade.
Looking at the GKE module (and Google docs), we actually can't seem to avoid automatic upgrades of masters anymore. The terraform min_master_version
we use to set master versions, is as it says, the minimum master version. The docs here seem to indicate we can do fuzzy matching, which might be preferred to avoid confusion, as currently our terraform says the cluster is one version, but in reality it is another.
I think the time might have come for us to consider fully embracing master auto upgrades, (although not sure if we have a choice), and we should update our terraform cluster versions to just pin to minor version (e.g. 1.16
) instead. This will at least hopefully minimise confusion. Though we should probably specify maintenance windows so these operations happen at a safe time.
Unfortunately pinning to a minor version is needed as we have had cases where issues with Gitlab chart (and depreciations etc) have prevented us from immediately upgrading.
As for node pools, they are still at a pinned version with auto upgrade disabled completely, however it is feasible to consider enabling some kind of targeted auto upgrade for them as well maybe.