Move GitLab's own secrets into Vault
Problem to solve
GitLab has a number of important secrets that it tracks itself; Kubernetes cluster credentials, credentials for its own database, and so on. We can reduce the attack surface area for these by moving them into an embedded Vault.
Permissions and Security
...I'm somewhat concerned/unsure whether it will be a good idea to colocate customer sensitive data in the same Vault as GitLab application secrets.
My primary concern is due to the OSS active/passive HA solution for Vault. In a large GitLab instance, if we use Vault for secrets, the availability of Vault will likely be critical to various components being able to start up. Consider that Unicorn workers and Sidekiq processes occasionally recycle and would need to reload secrets from Vault. In this case, what if the customer's usage of Vault for their own secrets is overwhelming the Vault primary and the GitLab components are not able to get their secrets in a timely manner?
This is completely theoretical. I have no idea how many queries a Vault primary can actually sustain. But I think it's important for us to at least think about this as we move forward.
Similarly, we should remember that Consul is the backend for Vault. In Omnibus HA environments, Consul is critical to watching and managing DB failover. I'm not sure if it's possible for Vault to overwhelm a Consul cluster but it would be bad if a customer's usage of Vault led to Consul problems.
Should we consider allowing Omnibus to deploy two completely separate Consul/Vault systems? This would also make it easier for a customer to 'bring your own' for the portion where they want to store their own secrets. We would then keep the Omnibus one for our own secrets.