GPG keys for vault

@gl-infra, I need your public keys to embed into our Vault setup from the very beginning.

What

Non ascii armored output of gpg --export $YOURKEYID, base 64 encoded. That's it.

Why

When initializing the encrypted storage backend, vault will generate a secret to encrypt data at rest and split it into number of chunks using Shamir Secret Sharing algorithm. Those chunks will be encrypted with each of your public GPG keys and then distributed to their respective owners. It will be done once per backend, which would be about the number of vaults, plus few testing one's, plus the number of times I screw up the tests and decide to start over . Therefore, set below whether you want to participate in CI/test vault unseals, or prefer only to be bothered with staging/production/infrastructure ones. Hashicorp's docs on subject if you're interested.

Scope of impact

For now, I recommend generating separate key with 1 year validity for vault explicitly, to keep things nice, tidy and separated from other keys before we have a defined procedure for all this, including key rotation. However, if you have objections to that, then I assume you know GPG well enough to have an established process of managing your keyrings, so do whatever you seem reasonable, as long as the email in the uid field of the sent public key ends up in @gitlab.com -- the reason for that being that I plan to automatically distribute the encrypted secrets as soon as they are generated.

How

It will work with any version of GPG, but I recommend 2.2 because of those ECC goodies :) Anyways, here's two instructions generating both ECC and RSA keys for vault (obviously have a badass passphrases!):

You can choose to create either an ECC key or an RSA key.

ECC

Awesome documentation if you wanna dig in.

1. Run gpg --expert --full-gen-key
2. Input 9 (ECC and ECC) on kind of key question
3. Input 1 (Curve 25519) selecting curve type
4. important Insert 1y to have 1 year expiration so we don't have eternal keys by default
5. important Insert something distinct for uid, I did this (make sure to use @gitlab.com email):

Real name: Ilya Frolov
Email address: ilya+vault@gitlab.com
Comment: vault unseal key
You selected this USER-ID:
    "Ilya Frolov (vault unseal key) <ilya+vault@gitlab.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

You'll get output similar to this:

pub   ed25519 2017-09-23 [SC] [expires: 2018-09-23]
      D21AF963D53F3121E3FA23D910FFA0D5EEC4824B
uid                      Ilya Frolov (vault unseal key) <ilya+vault@gitlab.com>
sub   cv25519 2017-09-23 [E] [expires: 2018-09-23]

Do gpg --export D21AF963D53F3121E3FA23D910FFA0D5EEC4824B | base64 > $yourname.asc and send MR with this file here. Congrats, mark yourself as done and whether you want to participate in test/CI unseals, which might be a bit of distraction while its taking shape.

RSA

Same thing basically, just larger keys and longer the generation:

1. Run gpg --full-generate-key
2. Select 1 (RSA and RSA) option
3. important Input at least 2048 bits for key length. No need to go higher, provided you set expiration. We'll rotate those keys soon.
4. important Input 1y as expiration data
5. Input meaningful UID, I did this:

Real name: Ilya Frolov
Email address: ilya+vault@gitlab.com
Comment: vault unseal key
You selected this USER-ID:
    "Ilya Frolov (vault unseal key) <ilya+vault@gitlab.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

Same as above, do gpg --export $keyid | base64 > $yourname.asc and send MR with this file here. Congrats, mark yourself as done and whether you want to participate in test/CI unseals, which might be a bit of distraction while its taking shape.

Progress tracking

Section for marking as done, adding people, removing people, etc.

• @ilyaf:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @ahanselka:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @ahmadsherif:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @eReGeBe:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @jarv:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @jtevnan:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @northrup:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @omame:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @pcarranza:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff
• @vlopez:
  • Keys in Place
  • Willing to be ping to unseal testing/CI stuff

mentioned in issue #1864 (closed)

changed the description

@ilyaf what does the process look like for rotating a key that's been used for sealing the vault? Say someone forgets or compromises their key? (I also added your check boxes )

Oh right, you totally can do those vertically

Good point on rotating the key, @northrup, thanks! Currently I'm not much worried about this particular set of keys (unless we lose all of them at the same time and w/o us knowing), since we don't have a production data yet, and -- provided they are 2048bit+, with 1y validity, independent of main keys we use for signing, and having nice passphrases -- I'm not worried much about some of them leaking too. I think losing/forgetting/compromising one of those will mean we can just regenerate and replace it.

However, I didn't think about the situation when we have a permanent storage with a lot of production data and then need to rotate the key that unseals it, and re-split it into parts. This will definitely need some attention. I think I can look into it as I switch the test/CI vaults secret backend from inmem to something persistent like S3, and then test the scenario of how we can change the set of keys while keeping vault contents at the same time.

Please not S3. I’d prefer the scenario of private git repo that we dump our consul out to over AWS S3.

My concern on rotating keys is that we make it possible to rotate out a single gpg key or rekey the whole thing without losing data or making it a big todo.

I was thinking about S3 for those vaults only: https://gitlab.com/gitlab-pkg/gitlab-vault/pipelines -- each kitchen job basically provisions new vault instance on DO now and then run inpec tests against it. I was thinking that S3 would be the easiest way to have some persistence across those runs, and also test the compatibility, that, say, next package version can use the backed of previous one. I'm not fully sold on "Only S3" idea, sure, its just feels easier now at the very early stages. We can use local fs and upload it as artifact even. As for staging vault and others (prod, infra, etc), they will use consul for HA as soon as I bootstrap the CA.

make it possible to rotate out a single gpg key or rekey the whole thing without losing data or making it a big todo.

Agreed, that should be thought about before we roll the production vaults.

@gl-infra please! let's get this done! (me first)

marked the checklist item Keys in Place as completed

marked the checklist item Willing to be ping to unseal testing/CI stuff as completed

marked the checklist item Keys in Place as completed

Also, since you're all subscribed to this thread now anyways, note that you'll need vault binary to be installed on your local machines to communicate with vault servers. The easiest way would be to reuse the procedure from test/CI pipelines which is defined here and can be called like this (assumes you have ~/bin, please set correct path if not):

wget -c https://gitlab.com/gitlab-com/gitlab-com-infrastructure/raw/master/Makefile && \
VA_INSTALL_TO="~/bin/" VA_VERSION=0.8.3 make vainstall && \
rm -f Makefile

This will also import hashicorp's public key into your key store if its not there already. 0.8.3 is the latest version as of now.

marked the checklist item Willing to be ping to unseal testing/CI stuff as completed

Or - you know, since we all compute with fruit ... brew install vault

@ilyaf or @northrup Is the idea here that we either generate ECC or RSA and is one preferred over the other?

I'm not sure how brew checks signatures, sorry, so I'm not recommending it myself

@jarv no, the general idea is that we just have keys, and start using them. Vault is just the perfect opportunity to do so as it requires them in order to securely distribute tokens. You can use either type, or both, just have some sensible defaults. In fact, fiddling around with keys, generating non-standard ones, losing the private parts and all sorts of reckless behaviour is encouraged at this stage. The more we account for, the better the end processes will be.

changed the description

@ilyaf thanks, I clarified this in the instructions.

marked the checklist item Keys in Place as completed

marked the checklist item Willing to be ping to unseal testing/CI stuff as completed

marked the checklist item Keys in Place as completed

marked the checklist item Willing to be ping to unseal testing/CI stuff as completed

marked the checklist item Keys in Place as completed

marked the checklist item Keys in Place as completed

marked the checklist item Willing to be ping to unseal testing/CI stuff as completed

marked the checklist item Keys in Place as completed

mentioned in merge request gitlab-pkg/gitlab-vault!12 (merged)

marked the checklist item Keys in Place as completed

marked the checklist item Willing to be ping to unseal testing/CI stuff as completed

marked the checklist item Keys in Place as completed

marked the checklist item Willing to be ping to unseal testing/CI stuff as completed

changed milestone to %WoW ending 2017-10-10

Awesome, thanks everyone, all keys are in place!

closed