...
  View open merge request
Commits (3)
......@@ -243,6 +243,9 @@ During an incident there are at least 2 roles, and one more optional
* [General Troubleshooting](troubleshooting/chef.md)
* [Error executing action `create` on resource 'directory[/some/path]'](troubleshooting/stale-file-handles.md)
### Certificates
* [Certificate runbooks](certificates/index.md)
## Learning
### Alerting and monitoring
......
## Chef Server
### Replacement
- Obtain the new certificate from [SSMLate](https://sslmate.com/console/orders/).
- ssh to `chef.gitlab.com`
- Create backup of the certificate (replacing 2019 with whatever year the old certificate started in)
```bash
sudo cp /etc/ssl/chef.gitlab.com.crt{,.2019}
```
- Copy the new certificate to the server as `/etc/ssl/chef.gitlab.com.crt` and change the permissions to `400` and owner to `root:root`
- `sudo chef-server-ctl hup nginx`
- Done!
### Rollback of a replacement
Sometimes stuff goes wrong. Good thing we made a backup! :)
- move the new certificate in a safe place
- restore the old certificate by renaming or copying it back.
- `sudo chef-server-ctl hup nginx`
- Done!
## Chef Vault
### Replacement
Make sure you know the data bag (e.g. `about-gitlab-com`) item (e.g. `_default`) and eventual fields (if they differ from `ssl_certificate` and `ssl_key`). Refer to the certificate table for that information.
- Obtain the new certificate from [SSMLate](https://sslmate.com/console/orders/).
- Create a local backup of the databag, by executing
```bash
knife vault show -Fj ${data_bag} ${item} > ${data_bag}_bak.json
```
- Format the new certificate (and/or key) to fit into json properly and copy the output to the clipboard. (The following command is executed with GNU sed)
```bash
sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' ${new_certificate}.pem
```
- Update the values in the data bag. Make sure to only edit the fields that were specified. Some data bags will contain multiple certificates!
```bash
knife vault edit ${data_bag} ${item}
```
- This should give you an error if the new data bag is not proper json. Still you should validate that by running `knife vault show -Fj ${data_bag} ${item} | jq .`. If that runs successfully, you have successfully replaced the certificate! Congratulations!
- Finally trigger a chef-run on the affected node(s). This should happen automatically after a few minutes, but it is recommended to observe one chef-run manually.
### Rollback of a replacement
Sometimes stuff goes wrong. Good thing we made a backup! :)
- Copy the contents of `${data_bag}_bak.json` into your clipboard
- Update the values in the data bag. Clear out the whole write-buffer and paste the json you just copied.
```bash
knife vault edit ${data_bag} ${item}
```
- Done!
## Fastly
### Replacement
(Close to) all certificates we use with Fastly are managed by them, too.
Should there be a manually managed certificate, you can follow these steps:
- Obtain the new certificate from [SSMLate](https://sslmate.com/console/orders/).
- *You will not be able to obtain a backup from fastly!*
- Follow https://docs.fastly.com/en/guides/tls-key-and-certificate-replacement
## GKMS
### Replacement
Make sure you know the item (e.g. `frontend-loadbalancer gprd`) and fields (if they differ from `ssl_certificate` and `ssl_key`). Refer to the certificate table for that information.
- Obtain the new certificate from [SSMLate](https://sslmate.com/console/orders/).
- Create a local backup of the gkms-vault, by executing
```bash
./bin/gkms-vault-show ${item} > ${item}_bak.json
```
- Format the new certificate (and/or key) to fit into json properly and copy the output to the clipboard. (The following command is executed with GNU sed)
```bash
sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' ${new_certificate}.pem
```
- Update the values in the gkms-vault. Make sure to only edit the fields that were specified. Some data bags will contain multiple certificates!
```bash
./bin/gkms-vault-edit ${item}
```
- This should give you an error if the new gkms-vault is not proper json. Still you should validate that by running `./bin/gkms-vault-show ${item} | jq .`. If that runs successfully, you have successfully replaced the certificate! Congratulations!
- Finally trigger a chef-run on the affected node(s). This should happen automatically after a few minutes, but it is recommended to observe one chef-run manually.
### Rollback of a replacement
Sometimes stuff goes wrong. Good thing we made a backup! :)
- Copy the contents of `${item}_bak.json` into your clipboard
- Update the values in the gkms-vault. Clear out the whole write-buffer and paste the json you just copied.
```bash
knife vault edit ${data_bag} ${item}
```
- Done!
This diff is collapsed.