Split Terraform state for Gitaly and Patroni in Production/Staging
As a potential corrective action for production#15997 (closed) I am wondering if it would make sense to take the Production and Staging Terraform environment and split Gitaly or Databases off into its own Terraform environments (e.g. {gprd,gstg}-patroni
, {gprd,gstg}-gitaly
)
This would be a shorter term fix for a larger problem in how we manage Terraform, but could be a quick way to isolate some of our critical infra from changes.
Pros
- Gitaly frequently is making configuration changes to add new nodes, these changes would be nice to isolate these.
- DBRES are focused on the db and this will help keep that focus
- Both Gitaly and Database do not have too many state dependencies on other infra, just network.
- Smaller state file
Cons
- We would need to do a datalookup for the vpc network
- Maybe we would need to do other datalookups
- We don't yet have the concept of two Terraform envs for the same GCP project
- Some variables would be duplicated across these envs
@gsgl @f_santos @nduff @pguinoiseau @rehab @sxuereb I'm sure this is something we have brought up before, do you know if we have any historical conversations about it or anything additional we might want to consider?
Edited by John Jarvis