2021-11-04: Update dev.gitlab.org certificates

Production Change

Change Summary

dev.gitlab.org's certificate is about to expire, we need to update its certificate to the renewed one. Related to #5863 (closed)

Change Details

  1. Services Impacted - ServiceNGINX
  2. Change Technician - @ahmadsherif
  3. Change Reviewer - @igorwwwwwwwwwwwwwwwwwwww
  4. Time tracking - 10 minutes minutes
  5. Downtime Component - none

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • Set label changein-progress on this issue
  • Make sure that you have sslmate installed on your workstation and that's linked to Ops SSLMate account (credentials are in 1Password).

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • On your workstation, download the new certificates
    • sslmate download dev.gitlab.org
  • Make sure the certificates are the renewed ones
    • openssl x509 -in dev.gitlab.org.chained.crt -dates -noout | grep notAfter # => notAfter=Nov 10 23:59:59 2022 GMT
  • Obtain the modulus of the key on the server
    • ssh dev.gitlab.org 'sudo openssl rsa -in /etc/gitlab/ssl/nginx.key -modulus -noout'
  • Obtain the modulus of the renewed certificate
    • openssl x509 -in dev.gitlab.org.chained.crt -modulus -noout
  • Make sure both moduli match
  • Make sure the certificates web and registry on the server are the same
    • ssh dev.gitlab.org 'sudo md5sum /etc/gitlab/ssl/nginx.crt /etc/gitlab/ssl/registry.crt'
  • Make sure the private key on the server of both certificates are the same
    • ssh dev.gitlab.org 'sudo md5sum /etc/gitlab/ssl/nginx.key /etc/gitlab/ssl/registry.key'
  • Store a backup of the vault contents
    • knife vault show dev-gitlab-org _default --format=json > tmp-vault.json
  • Copy the "inlined" certificate to your clipboard
    • cat dev.gitlab.org.chained.crt | sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' | xclip
  • Edit the values under the keys omnibus-gitlab.ssl.certificate and omnibus-gitlab.ssl.registry_certificate with your clipboard content
    • knife vault edit dev-gitlab-org _default
  • Run Chef on the server
    • ssh dev.gitlab.org 'sudo chef-client'

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • In your browser, check if the certificate of dev.gitlab.org has a new expiry date
  • Confirm that the certificates on the server have been updated
    • ssh dev.gitlab.org 'sudo openssl x509 -in /etc/gitlab/ssl/nginx.crt -dates -noout | grep notAfter' # => notAfter=Nov 10 23:59:59 2022 GMT
    • ssh dev.gitlab.org 'sudo openssl x509 -in /etc/gitlab/ssl/registry.crt -dates -noout | grep notAfter' # => notAfter=Nov 10 23:59:59 2022 GMT
  • Remove the vault backup
    • rm tmp-vault.json

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes

  • Use tmp-vault.json to revert the certificate values to the old ones`
    • knife vault edit dev-gitlab-org _default
  • - Run Chef on the server
    • ssh dev.gitlab.org 'sudo chef-client'

Monitoring

Key metrics to observe

  • Metric: Metric Name
    • Location: Dashboard URL
    • What changes to this metric should prompt a rollback: Describe Changes

Summary of infrastructure changes

  • Does this change introduce new compute instances?
  • Does this change re-size any existing compute instances?
  • Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?

Summary of the above

Changes checklist

  • This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
  • This issue has the change technician as the assignee.
  • Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Necessary approvals have been completed based on the Change Management Workflow.
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents.
Edited by Ahmad Sherif