Track and upgrade GKE to version 1.14.x
Due to a bug in the implementation of Kubernetes limits in version 1.13.x, CPU limits of 1
== 1000m
, however when attempting to detect the value, it is seen as different. When an operation attempts to change from 1 to 1000m are attempted, Kubernetes will respond with an error that nothing was changed. There's 2 issues here:
- Whatever is detecting the limit is not properly handling a situation where 1 and 1000m are the same and attempts to change something unnecessarily
- The Kubernetes API is correct in that it's not going to change anything, but it shouldn't be an error.
This is coming from the fluentd component of GKE. This is something that is mostly out of our control and a fix was built into the Kubernetes API in version 1.14. More context can be found here: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204637830
Utilize this issue to perform the following:
-
Wait till GKE release version 1.14 - currently it's on the Rapid channel, and can only be installed in an alpha cluster which we must not do - One can monitor the releases of GKE here: https://cloud.google.com/kubernetes-engine/docs/release-notes
-
Upgrade our clusters in a slow controlled fashion