Prepare for GKE 1.24 Upgrade which deprecates dockershim entirely

We have received the following email from Google support outlining the fact that we (Gitlab) have some work to do in order to prepare for the GKE 1.24 upgrade coming later this year. This is because the old "dockershim" method of the Kubelet managing containers is deprecated entirely in favor of using containerd.

Contents of the email

 Support for Docker as a container runtime on Kubernetes nodes will be removed from OSS Kubernetes and GKE starting with v1.24. Please migrate your GKE workloads to Containerd as soon as possible.

Hello Google Kubernetes Engine Customer,

We’re writing to let you know that OSS Kubernetes is moving on from Dockershim and removing built-in support for the Docker container runtime (“Dockershim”) starting with v1.24. For this reason, GKE will not support GKE node images based on Dockershim starting with GKE v1.24.

Most user workloads do not have dependencies on the container runtime. If you use a node image based on Dockershim, please migrate your GKE workloads to a Containerd node image as soon as possible.

For your reference, below are the GKE node images for the Containerd and Docker container runtimes, respectively:

    Containerd container runtime (recommended): cos_containerd, ubuntu_containerd, windows_ltsc_containerd, windows_sac_containerd
    Docker container runtime (unsupported starting with v1.24): cos, ubuntu, windows_ltsc, windows_sac

What do I need to know?

GKE will remove support for Dockershim starting with GKE v1.24. To mitigate the impact of this change, we want to help you transition from Docker to Containerd node images:

    New node pools:

    You will not be able to create new node pools that use a Docker node image —starting with GKE v1.23— when:
        Creating a new cluster,
        Adding a node pool to an existing cluster, or
        Using “Node Auto-provisioning” (NAP) with --autoprovisioning-image-type set to Docker node images.
        For existing clusters, you will also not be able to change the value of --autoprovisioning-image-type to Docker node images.

    Upgrading to GKE v1.23:

    If you are upgrading your GKE clusters from GKE v1.22 to v1.23, then you will be able to continue using:
        Docker node pools that were configured before the upgrade.
        Cluster Autoscaler on Docker node pools.
        Node Auto-provisioning (NAP) with --autoprovisioning-image-type set to Docker node images if it was configured before upgrading to v1.23

    However, we highly recommend you to migrate to GKE node images that use the Containerd container runtime.

    Upgrading to GKE v1.24 (when available):

    If you are upgrading your GKE clusters from GKE v1.23 to v1.24, and if your cluster contains one or more node pools running Docker node images, then your control plane and your nodes will be blocked from upgrading to GKE v1.24.

    Once you have migrated all node pools to Containerd, then your control plane and nodes will be unblocked and will continue upgrading to GKE v1.24.
    This applies whether your cluster is enabled for auto upgrade or you are upgrading it manually.

    Support and assistance:

    We understand that your situation might be unique, so our team is available to assist you throughout this journey. If you encounter any issues, we encourage you to contact Google Cloud support.

You can continue to use docker build or docker push as part of your development cycle or CI/CD pipeline with Containerd-based GKE node images. This change affects only the way containers are started and stopped via the Docker container runtime.

The projects listed below are currently using Docker container runtime:

    service-staging-55dd44
    cust-production-e9b9ab
    gemnasium-staging
    gemnasium-production
    gs-production-efd5e8
    gitlab-staging-1
    license-prd-bfe85b
    gitlab-org-ci-0d24e2
    service-master-84ef28
    license-stg-8ba766
    gs-staging-23019d
    master-b0afce
    cluster-resize-76e62d
    cust-staging-4bed71
    service-prod-c2bbd3

Containerd is a Container Runtime Interface (CRI) compliant container runtime, and is the default runtime on GKE. The migration to Containerd only affects the way containers are managed on nodes. Support for Docker container images and the docker tool often used to build them will continue unchanged.
What do I need to do?

Migrate your GKE workloads and associated tooling to Containerd as soon as possible so that you are not blocked from upgrading to GKE v1.24 and do not run into issues during the GKE v1.23 Dockershim end-of-life.

Migrate your GKE node pools from Docker to Containerd node images through this 3-step process:

    Identify: use the sample script to iterate over all your projects and identify node pools (and workloads) that are using Docker node images.
    Verify: deploy the identified workloads on test node pools running Containerd node images either in a staging environment or canary clusters to ensure there are no issues. We encourage you to contact Google Cloud support, if needed.
    Migrate: your node pools to Containerd using by migrating workloads to different machine types and updating your node image-type. If you don’t migrate to Containerd by v1.23, you will be blocked from upgrading to GKE v1.24.

If you don’t have a plan to migrate to Containerd in time, you might have last-minute blocking issues.
Need more assistance?

    You can find the list of differences of Containerd images and advice on mitigating these differences in the GKE documentation page.
    Please review the list of known issues. The list is updated frequently as we fix issues.

If you have any questions or require assistance, please reply to this email to contact Google Cloud Support.

There is a script that Google have supplied to help finding problem node pools easier.

This issue will be marked complete when the projects in the above list (that we care about) are confirmed to have no node pools using docker-shim.