Convert GKE thanos objstore.yaml to use GKE Workload Identity
Currently in order to configure our thanos objstore.yaml for the thanos sidecar in all our GKE clusters, we have to setup a Kubernetes secret that is populated with the entire contents of a GCP Service account Key in JSON format. As you can only obtain a copy of this content (including the private key) when the Key is created, we resort to having to store these credentials in 1password and manually populating secrets with this data as needed.
In order to allow greater flexibility, minimise manual steps, and increase security, we want to get rid of this manual process of copying around the contents of this service key.
To do this, we want to leverage GKEs workload identity functionality, unique to GKE installations. Basically this will allow us to add annotations to the pods/containers that map that pod/container to a particular service account in GCP. This means that pod/container will have permissions to GCP as if it had a copy of a Service Key for that service account.
For this particular use case, we wish to remove the service_account:
stanza from the thanos-objstore-config
Kubernetes secret object, and instead apply some configuration through terraform and then helm to map the thanos-sidecar container to that service account instead. This will allow us to completely remove that service accounts keys from being involved in any Kubernetes setup at all.
What needs to be done
-
Modify the gke
terraform module to allow us to enable workload identity on GKE clusters -
Enable workload identity on all our GKE clusters -
Modify the storage-buckets
terraform module to add a IAM binding between the kubernetes service account thanos uses and the GCP service account we want to use e.g.
resource "google_service_account_iam_member" "service-account" {
role = "roles/iam.workloadIdentityUser"
service_account_id = "projects/gitlab-pre/serviceAccounts/pre-prometheus-sa@gitlab-pre.iam.gserviceaccount.com"
member = "serviceAccount:gitlab-pre.svc.id.goog[monitoring/gitlab-monitoring-promethe-prometheus]"
}
-
Modify the helm deployment of prometheus with the thanos sidecar to make sure the following annotation is added
iam.gke.io/gcp-service-account: "pre-prometheus-sa@gitlab-pre.iam.gserviceaccount.com"