Certificate automation for workspaces.gitlab.dev
We had a recent incident where it was discovered that workspaces.gitlab.dev
has a manual process for updating certificates, as well as no alert or monitoring in place to warn of an upcoming certificate expiry.
We would ideally need two outcomes for this corrective action
- Ensure
workspaces.gitlab.dev
has its SSL expire monitored. Ideally routed to the DRIs of the system, rather than oncall. - Automate the certificate renewal prcoess.
Currently the certificate is generated from SSLMate
and is expected to be uploaded to Google Secrets Manager. The terraform that pulls that secret is here https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure/-/blob/main/remote-development/gitlab_workspaces_proxy.tf. So even with an updated certificate it would require a terraform run to update the certificate.
We have a few options available for automating this process including our certificate-updater. However since this is a GKE service deployed through helm, the better solution is to move the certificate generation to cert-manager using LetsEncrypt.
Related Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17992
Details
- Point of contact for this request: @user
- If a call is needed, what is the proposed date and time of the call: Date and Time
- Additional call details (format, type of call): additional details
SRE Support Needed Support Request Details