All attempts to create GKE Cluster Fail with Invalid Request

Summary

All attempts to create a GKE cluster through GitLab.com currently fail with an error:

Something went wrong while creating your Kubernetes cluster on Google Kubernetes Engine
Failed to request to CloudPlatform; Invalid request

The Logs Explorer in Google reports simply reports ERROR, without any meaningful information and error message. I can manually create a cluster with the same zone, nodes count and cluster name in the same GCP project.

• Internal Slack discusstion thread.
• Internal Customer ticket.

Steps to reproduce

Follow the steps in our Creating the Cluster on GKE guide on any project or a group on GitLab.com

Example Project

https://gitlab.com/gitlab-gold/kategrechishkinatestgroup/213437-install-gke-cluster/-/clusters/150126

What is the current bug behavior?

Unable to create GKE cluster through GitLab

What is the expected correct behavior?

The cluster should be created successfully.

Relevant logs and/or screenshots

Logs from Google Log Exporter:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 3
    },
    "authenticationInfo": {
      "principalEmail": "user@example.com",
      "principalSubject": "user:user@example.com"
    },
    "requestMetadata": {
      "callerIp": "gce-internal-ip",
      "callerSuppliedUserAgent": "GitLab/13.12 (GPN:GitLab;)",
      "requestAttributes": {
        "time": "2021-05-19T08:50:48.313942172Z",
        "auth": {}
      },
      "destinationAttributes": {}
    },
    "serviceName": "container.googleapis.com",
    "methodName": "google.container.v1beta1.ClusterManager.CreateCluster",
    "authorizationInfo": [
      {
        "permission": "container.clusters.create",
        "granted": true,
        "resourceAttributes": {}
      }
    ],
    "resourceName": "projects/support-testing-168620/zones/us-central1-a/clusters/testing-creation",
    "request": {
      "cluster": {
        "legacyAbac": {},
        "ipAllocationPolicy": {
          "clusterIpv4CidrBlock": "/16",
          "useIpAliases": true
        },
        "addonsConfig": {},
        "name": "testing-creation"
      },
      "@type": "type.googleapis.com/google.container.v1alpha1.CreateClusterRequest"
    },
    "response": {
      "@type": "type.googleapis.com/google.container.v1alpha1.Operation"
    },
    "resourceLocation": {
      "currentLocations": [
        "us-central1-a"
      ]
    }
  },
  "insertId": "ut2h2idfvf1",
  "resource": {
    "type": "gke_cluster",
    "labels": {
      "cluster_name": "testing-creation",
      "project_id": "support-testing-168620",
      "location": "us-central1-a"
    }
  },
  "timestamp": "2021-05-19T08:50:48.552634267Z",
  "severity": "ERROR",
  "logName": "projects/support-testing-168620/logs/cloudaudit.googleapis.com%2Factivity",
  "receiveTimestamp": "2021-05-19T08:50:48.870792116Z"
}

Output of checks

This bug happens on GitLab.com

Possible fixes

The quick fix is to pin the GKE version to 1.18 on the API call to create the cluster.

Longer term we need to stop using Basic Auth.

See details.