Do not delete GSM secret revisions
We ran into an issue with Google Secret Manager secret revisions being marked as deleted as part of the terraform replace operation, resulting in containers failing to start with this message:
Failed to access version "latest" of Secret Manager secret "ai-gateway-l0ou-otel-yml" for file config.yaml: generic::not_found: Secret [projects/39671935963/secrets/ai-gateway-l0ou-otel-yml] not found or has no versions.
We see a failed pipeline that resulted in the old secret revision being deleted. This broke the existing deployment. I suspect this may be happening during each deployment, the old Cloud Run revision becoming invalid.
When looking at the actual secrets revisions in GSM, we see the old revision in a destroyed state:
We may want to prevent old revisions from being destroyed altogether. Not sure if we can easily do this while using a single google_secret_manager_secret_version
resource in terraform.