Docs: clarify Gitaly server configuration requirements in mixed environments
Problem to solve
- Make the Gitaly documentation clearer about how to add Gitaly cluster to existing environments.
- Address some apparent inconsistencies
- Proactively add documentation for issues which customers are more likely to encounter migrating to Gitaly cluster.
Two statements about server (gitaly-to-gitaly) connectivity in the docs raise questions:
- Under client configuration
Gitaly makes the following assumptions: Your [..] Gitaly servers can reach each other.
- Under client configuration: mixed configuration
All addresses must be reachable from the other Gitaly servers.
Except for the mixed configuration, which is both client and server configuration on a Omnibus installation, none of the guides to Gitaly server configuration include configuration which is explicit about how Gitaly servers reach each other. [1] Is this because it's not necessary to be explicit? See [6] below as well.
Further details
Support for running Gitaly on NFS will be removed in GitLab 14.0.
A common pattern will be that customers construct a Gitaly cluster, add it to their existing GitLab environment, and may then migrate to it, using the repository moves API.
Clarification of the main question was requested by a customer via a support ticket (internal links).
Here's some potential existing implementations of Gitaly. Customers might add a gitaly cluster to any one or these, or they may already be running a mixture of these, and want to add Gitaly cluster as a third Gitaly back end.
- One combined GitLab server, with Gitaly on a unix socket.
- Multiple combined GitLab servers, each running Gitaly on a unix socket, sharing data via NFS.
- Two GitLab servers: one dedicated to Gitaly, one running the rest of the stack. Gitaly is accessed via TCP.
- A multi-server setup with two or more dedicated Gitaly servers sharing data via NFS. Gitaly over TCP.
Complicated Gitaly setups are covered under the client configuration section of the Gitaly docs.
For reference below:
On gitaly1.internal:
git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data' }, 'storage1' => { 'path' => '/mnt/gitlab/git-data' }, }) On gitaly2.internal: git_data_dirs({ 'storage2' => { 'path' => '/srv/gitlab/git-data' }, })
There seems to be an inconsistency:
Your gitaly1.internal Gitaly server can be reached at gitaly1.internal:8075 from your Gitaly clients, and that Gitaly server can read, write, and set permissions on /mnt/gitlab/default and /mnt/gitlab/storage1.
Your gitaly2.internal Gitaly server can be reached at gitaly2.internal:8075 from your Gitaly clients, and that Gitaly server can read, write, and set permissions on /mnt/gitlab/storage2.
[2] Should the paths in the server section, quoted above, and this client section be the same?
storage | server docs | client docs |
---|---|---|
default@gitaly1.internal |
/var/opt/gitlab/git-data |
/mnt/gitlab/default |
storage1@gitaly1.internal |
/mnt/gitlab/git-data |
/mnt/gitlab/storage1 |
storage2@gitaly2.internal |
/srv/gitlab/git-data |
/mnt/gitlab/storage2 |
I wonder if the server paths were revised for the reference architectures, but the Gitaly client docs not kept in sync.
Also in the client section:
Gitaly makes the following assumptions:
- Your
gitaly1.internal
andgitaly2.internal
Gitaly servers can reach each other.You can’t define Gitaly servers with some as a local Gitaly server (without
gitaly_address
) and some as remote server (withgitaly_address
) unless you use mixed configuration.
[3] Last sentence: does this need fixing? mixed configuration doesn't support servers without gitaly_address
either.
GitLab can reside on the same server as one of many Gitaly servers, but doesn’t support configuration that mixes local and remote configuration. The following setup is incorrect, because:
All addresses must be reachable from the other Gitaly servers.
storage1
is assigned a Unix socket forgitaly_address
which is invalid for some of the Gitaly servers.
[4] How exactly does Gitaly-to-Gitaly communication work?
- The documentation states that the Gitaly servers must be able to reach each other. Exposing ports to the network is one step, but if an environment had two, three, or more discrete Gitaly implementations, how do they know where they all are so they can reach each other?
One storage configuration entry is required: default
.
- citation: gitlab#36175
- if customers do migrate from their existing Gitaly to Gitaly cluster, at some point they'll want to shut down their old Gitaly and remove
default
from their configuration. - This doesn't work. Documentation about extending their environment to include Cluster should encourage customers to plan ahead on this.
[5] Should we document this?
The reference architecture docs as well as the configure Gitaly servers section provide examples of configuring multiple Gitaly servers in a sharding configuration
[6]Is sharding actually a better way to think about environments with multiple gitaly back ends?
- The examples there do not have references on each server to other servers' storage
- This would seem to suggest that nothing in fact needs to be done on the Gitaly servers so they can reach each other.
On gitaly1.internal:
git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data' }, 'storage1' => { 'path' => '/mnt/gitlab/git-data' }, })
On gitaly2.internal:
git_data_dirs({ 'storage2' => { 'path' => '/mnt/gitlab/git-data' }, })
Proposal
Who can address the issue
I can raise an MR to clarify the docs.