Limit access to GCP projects for users access through the UI
Problem
At the moment anyone both infrastructure and certain groups has Owner permissions for certain/all GCP projects. This gives a lot of flexibility for SRES/certain engineers to operate the infrastructure however this can also be dangerous because humans make mistakes and an innocent change that effects production. For full context, this came from a discussion in production#5489 (comment 673517531) where exactly this happened. Removing permissions is never fun, and I don't want this to sounds like a punishment to anyone, but the idea is to protect the users from doing mistakes. It's the same reason why you shouldn't run your computer as root
at all times.
Solution
Update permissions for SREs/engineers to not be an owner level, but something more locked down, where they can update project-level settings or do any changes.
This will have the following benefits
- It's safer to browse the project.
- Any changes are forced to be done through terraform, so you can actually get a diff.
- It might be easier to give permissions in the future.
- Follow the least privileged model.
Break glass
Now during incidents sometimes rollout out a change through terraform is not possible, or not ideal. This is why need to have a break glass Procedure in case this happens. One possible solution is for each SRE to have two accounts like one with locked down permissions and one with owner permissions similar to what we have on GitLab.com, where their daily driver is a normal GitLab user, but then they have a separate logging for the admin user.