Validate service account permissions at cluster creation

Problem to solve

Originally discussed in this thread.

When a cluster is added, a user is notified if GitLab is either unable to reach the provided URL or cannot authenticate to the Kubernetes API (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27403). However it is possible for both of these checks to succeed, but GitLab may still not have permission to create the resources required.

Resources are no longer created as soon as a cluster is added (gitlab-ce#57115), so we would need a mechanism that checks permissions without actually creating any resources. From the linked thread:

there doesn't seem to be a straight-forward way to check if a token has cluster-admin or not. We only have the token, and would need some customisation to even get name of the service account (see https://github.com/kubernetes/kubernetes/issues/30784 and https://github.com/kubernetes/kubernetes/pull/30829). There is also can-i (https://kubernetes.io/docs/reference/access-authn-authz/authorization/#checking-api-access), but we would would have to add support for it.

This check may prove to not be required, as insufficient permissions will cause a deployment CI job to fail with the same (or similar) error message (https://gitlab.com/gitlab-org/gitlab-ce/issues/54506).

Preview

status === 'service_account_failure' (cluster_status.json)

Further details

This is only relevant to managed clusters, as these are the only clusters we need to create service accounts for.

Links / references

gitlab-ce#55447 / gitlab-ce!27403,

https://kubernetes.io/docs/reference/access-authn-authz/authorization/#checking-api-access