Proper procedure to replace Gitaly nodes.
Support Request for the Gitaly Team
The goal is to keep these requests public. However, if customer information is required to the support request, please be sure to mark this issue as confidential.
This request template is part of Gitaly Team's intake process.
Customer Information
S500PL000002Q3MAYA0
https://gitlab.zendesk.com/agent/tickets/463388:
Installation Size:
Architecture Information:
Slack Channel:
Additional Information:
Support Request
Severity
Problem Description
Hi, team, A quick question. I have a customer that wants to replace 2 of their Gitaly nodes. This is the procedure that I gave.
- Install the new gitaly as per our documentation
- Copy
gitlab.rb
file and thegitlab-secrets
file to the new Gitlay server - Add the configuration of the two servers. In Gitaly configuration
- Add the configuration of the two servers. In Praefect configuration
- Reconfigured all Gitaly nodes and Praefect nodes
- Run the
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes
in praefect does not show any error - Ensure that you do not have any unavailable repositories. Run this command in praefect
/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -partially-unavailable
- Depending on your repository size, it may take a while for the replication manager to finish the replication of all repositories
- Before removing the old server. Switch them off and try to access all repositories
- if they are all accessible without issues, you can now remove the configuration again from the praefect
gitlab.rb
and Gitaly - Ensure that you reconfigure after changing
gitlab.rb
file
Correct me or let me know what I have missed.
I tried it on my test server, and I could replace the Gitaly nodes. However, I noted that the primary Gitaly in the repositories table does not change. These records reference an inexistence, Gitaly, since we have already removed them. Earlier, there was a customer who removed the Gitaly reference, and accessing the repositories that were removed caused error 500.
Troubleshooting Performed
What specifically do you need from the Gitaly team
Author Checklist
-
Customer information provided -
Severity realistically set -
Clearly articulated what is needed from the Gitaly team to support your request by filling out the What specifically do you need from the Gitaly team
/cc @mjwood @andrashorvath @jcaigitlab @john.mcdonnell @gerardo