Address fixes for terraform init problems after bad configuration
If the user runs provision_cluster.sh then a terraform init will attempt to provision the GCS buckets.
When this occurs the bucket name and prefix will be written to the Terraform state file at /repo/terraform/.terraform/terraform.tfstate.
However if the bucket and/or prefix are specified incorrectly on the first run, they will not be updated on subsequent runs.
The user will see a non-descriptive API error such as:
Error inspecting states in the "gcs" backend:
querying Cloud Storage failed: googleapi: Error 403: user@project.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket., forbidden
This leads to a large amount of time revisiting service account and bucket permissions.
By adding the -reconfigure parameter the state file will be ignored on terraform init as described in the terraform docs.
This means that the state file will be populated from secrets/gitlab-cluster-rc.sh everytime terraform_pretasks() is called.
Related to #168 (closed) (didn't see this was closed when I created the MR so some further discussion is probably required to avoid conflict with !29 (merged))
After some feedback in the threads below, we decided that adding the destructive -reconfigure flag was undesirable in the scope of Orchestrator. Instead, we can update the troubleshooting documentation regarding this error so end users can fix it themselves to ensure Orchestrator doesn't over-reach and that end users understand and own the deletion of information.