Move consul to the new VNET and make it secure

We need to move Consul to the new GitLabProd vnet and secure it. This means only allow access from the GitLabProd address space and the decommissioned vnets that still need it (not sure if any, probably not). We're still only using the kv store so it's easy enough.

My idea is to create a new 3-node cluster using A1 VMs and then copy the kv store contents.

It's important to encrypt gossip and make use of ACL tokens.

Points to discuss:

  • How do we then connect to it with to use Terraform?
    • I'm thinking of moving the state to S3. This would allow us to properly lock down Consul and have server-side versioning of the state.
  • How should we name these hosts? The current scheme is consulXX.kv.gitlab.com.
    • I personally don't like the kv part since we'll be mostly using consul for service discovery. We could use consulXX.be.gitlab.com or something more specific for support services (but not ss, please).

/cc @gl-infra

Edited by Daniele Valeriani [GitLab]