Clean up KubernetesNamespace records without service account tokens

Problem to solve

Currently it is possible for GitLab to persist Kubernetes namespace records without a service account token present. This is not ideal as when the time comes to deploy to the namespace there is no token to use, and extra logic has to be applied to figure out how to perform the deploy.

Proposal

Prerequisite

Assuming we've removed all possible ways of creating clusters_kubernetes_namespaces with encrypted_service_account_token=null then we want to set a null constraint. We need to confirm that this is the case before we can tackle this issue. We should wait some time and check:

select count(*) from clusters_kubernetes_namespaces where encrypted_service_account_token is null and created_at > now() - interval'1 week';

Plan

1. Investigate why we seem to be encountering so many of these records - https://gitlab.com/gitlab-org/gitlab-ce/issues/63324#note_181829749 is a possible theory.
2. Sanitise (attempt to populate token, and destroy record if unable) existing Clusters::KubernetesNamespace records.
3. Make clusters_kubernetes_namespaces.encrypted_service_account_token column non-nullable

What does success look like, and how can we measure that?

Success: clusters_kubernetes_namespaces.encrypted_service_account_token is non-nullable at the database level.

Links / references

This is related to https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29657 . Ideally that check could also be removed if the column was non-nullable.