Turn on Gitaly TLS on gitlab.com
For https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/6484
C1
Production Change - Criticality 1Change Objective | Turn on Gitaly TLS for gitlab.com |
---|---|
Change Type | ConfigurationChange |
Services Impacted | ServiceGit |
Change Team Members | @alejandro |
Change Criticality | C1 |
Change Reviewer | A colleague who will review the change |
Tested in staging | Yes https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3150 https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3151 |
Dry-run output | If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result |
Due Date | Date and time in UTC timezone for the execution of the change, if possible add the local timezone of the engineer executing the change |
Time tracking | To estimate and record times associated with changes ( including a possible rollback ) |
Downtime Component | No downtime, ideally |
Detailed steps for the change
-
Merge and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3147
-
Run chef-client on the file servers:
knife ssh roles:gprd-base-stor "sudo chef-client"
-
Verify gitaly is accepting TLS connections:
ssh console-01-sv-gprd.c.gitlab-production.internal
sudo gitlab-rails c
- Run the following script:
# Replace all shard address with the new addresses (don't worry, only in-memory for this console) Gitlab.config.repositories.storages.each { |k, v| Gitlab.config.repositories.storages[k][:gitaly_address] = v.gitaly_address.gsub(/tcp:\/\/(.+):9999/, 'tls://\1:9998') } Gitlab::GitalyClient.clear_stubs! # Health check gitaly Gitlab::HealthChecks::GitalyCheck.readiness # You should see success=true for all shards
-
Merge and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3148
-
Run chef-client with concurrency limits:
knife ssh -C 5 "roles:gprd-base-fe OR roles:gprd-base-be" "sudo chef-client"
Rollback steps
- Revert https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3148
- Revert https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3147
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
SRE on-call has been informed prior to change being rolled out -
There are currently no open issues labeled as ServiceMonitoring with severities of ~S1 or ~S2
Edited by Alejandro Rodríguez