Skip to content
Snippets Groups Projects

Update documentation to match Gitaly's new configuration structure

Merged Sami Hiltunen requested to merge smh-update-gitaly-config-structure into master
All threads resolved!
13 files
+ 571
308
Compare changes
  • Side-by-side
  • Inline
Files
13
  • Omnibus previously had a custom configuration structure for Gitaly
    that didn't match the actual structure of Gitaly. This made a number
    of things more difficult than they had to be. Omnibus was recently
    updated to place Gitaly's configuration under gitaly['configuration']
    hash which gets directly marshaled as TOML for the final Gitaly
    configuration. While there is still backwards compatibility handling
    in place, we should update the documentation to use the new structure
    in preparation for deprecating the old configuration keys. This commit
    updates Gitaly's documentation to refer to the new keys.
@@ -143,7 +143,13 @@ Gitaly and GitLab use two shared secrets for authentication:
To configure the _Gitaly token_, edit `/etc/gitlab/gitlab.rb`:
```ruby
gitaly['auth_token'] = 'abc123secret'
gitaly['configuration'] = {
# ...
auth: {
# ...
token: 'abc123secret',
},
}
```
There are two ways to configure the _GitLab Shell token_.
@@ -230,14 +236,21 @@ Updates to example must be made at:
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server.
gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
# Make Gitaly accept connections on all network interfaces. You must use
# firewalls to restrict access to this address/port.
# Comment out following line if you only want to support TLS connections
gitaly['listen_addr'] = "0.0.0.0:8075"
# Authentication token to ensure only authorized servers can communicate with
# Gitaly server
gitaly['auth_token'] = 'AUTH_TOKEN'
gitaly['configuration'] = {
# ...
#
# Make Gitaly accept connections on all network interfaces. You must use
# firewalls to restrict access to this address/port.
# Comment out following line if you only want to support TLS connections
listen_addr: '0.0.0.0:8075',
auth: {
# ...
#
# Authentication token to ensure only authorized servers can communicate with
# Gitaly server
token: 'AUTH_TOKEN',
},
}
```
1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server:
@@ -247,24 +260,33 @@ Updates to example must be made at:
On `gitaly1.internal`:
```ruby
git_data_dirs({
'default' => {
'path' => '/var/opt/gitlab/git-data'
},
'storage1' => {
'path' => '/mnt/gitlab/git-data'
},
})
gitaly['configuration'] = {
# ...
storage: [
{
name: 'default',
path: '/var/opt/gitlab/git-data',
},
{
name: 'storage1',
path: '/mnt/gitlab/git-data',
},
],
}
```
On `gitaly2.internal`:
```ruby
git_data_dirs({
'storage2' => {
'path' => '/srv/gitlab/git-data'
},
})
gitaly['configuration'] = {
# ...
storage: [
{
name: 'storage2',
path: '/srv/gitlab/git-data',
},
],
}
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -461,17 +483,28 @@ example:
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
# Address of the GitLab server that also has Gitaly running on it
'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' },
'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075' },
'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' },
})
# Make Gitaly accept connections on all network interfaces
gitaly['listen_addr'] = "0.0.0.0:8075"
# Or for TLS
gitaly['tls_listen_addr'] = "0.0.0.0:9999"
gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
gitaly['configuration'] = {
# ...
#
# Make Gitaly accept connections on all network interfaces
listen_addr: '0.0.0.0:8075',
# Or for TLS
tls_listen_addr: '0.0.0.0:9999',
tls: {
certificate_path: '/etc/gitlab/ssl/cert.pem',
key_path: '/etc/gitlab/ssl/key.pem',
},
storage: [
{
name: 'storage1',
path: '/mnt/gitlab/git-data',
},
],
}
```
`path` can be included only for storage shards on the local Gitaly server.
@@ -600,16 +633,21 @@ To configure Gitaly with TLS:
<!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab -->
```ruby
gitaly['tls_listen_addr'] = "0.0.0.0:9999"
gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
gitaly['configuration'] = {
# ...
tls_listen_addr: '0.0.0.0:9999',
tls: {
certificate_path: '/etc/gitlab/ssl/cert.pem',
key_path: '/etc/gitlab/ssl/key.pem',
},
}
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. Verify Gitaly traffic is being served over TLS by
[observing the types of Gitaly connections](#observe-type-of-gitaly-connections).
1. Optional. Improve security by:
1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in
1. Disabling non-TLS connections by commenting out or deleting `gitaly['configuration'][:listen_addr]` in
`/etc/gitlab/gitlab.rb`.
1. Saving the file.
1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -733,9 +771,16 @@ settings:
1. Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Default is 2 workers. The minimum is 2; 1 worker is always reserved as
# a passive stand-by.
gitaly['ruby_num_workers'] = 4
gitaly['configuration'] = {
# ...
'gitaly-ruby': {
# ...
#
# Default is 2 workers. The minimum is 2; 1 worker is always reserved as
# a passive stand-by.
num_workers: 4
},
}
```
1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -775,21 +820,23 @@ example:
```ruby
# in /etc/gitlab/gitlab.rb
gitaly['concurrency'] = [
{
'rpc' => "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
'max_per_repo' => 20,
'max_queue_time' => "1s",
'max_queue_size' => 10
},
{
'rpc' => "/gitaly.SSHService/SSHUploadPackWithSidechannel",
'max_per_repo' => 20
'max_queue_time' => "1s",
'max_queue_size' => 10
}
]
gitaly['configuration'] = {
# ...
concurrency: [
{
rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel',
max_per_repo: 20,
max_queue_time: '1s',
max_queue_size: 10,
},
{
rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel',
max_per_repo: 20,
max_queue_time: '1s',
max_queue_size: 10,
},
],
}
```
- `rpc` is the name of the RPC to set a concurrency limit for per repository.
@@ -864,40 +911,47 @@ When these limits are reached, performance may be reduced and users may be disco
> This method of configuring repository cgroups was introduced in GitLab 15.1.
To configure repository cgroups in Gitaly using the new method, use the following settings for the new configuration method
to `gitaly['cgroups']` in `/etc/gitlab/gitlab.rb`:
to `gitaly['configuration'][:cgroups]` in `/etc/gitlab/gitlab.rb`:
- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
- `mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
- `hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab
creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root`
when Gitaly starts.
- `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all
- `memory_bytes` is the total memory limit that is imposed collectively on all
Git processes that Gitaly spawns. 0 implies no limit.
- `cgroups_cpu_shares` is the CPU limit that is imposed collectively on all Git
- `cpu_shares` is the CPU limit that is imposed collectively on all Git
processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares,
which represents 100% of CPU.
- `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git
- `repositories.count` is the number of cgroups in the cgroups pool. Each time a new Git
command is spawned, Gitaly assigns it to one of these cgroups based
on the repository the command is for. A circular hashing algorithm assigns
Git commands to these cgroups, so a Git command for a repository is
always assigned to the same cgroup.
- `cgroups_repositories_memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
0 implies no limit. This value cannot exceed that of the top level `cgroups_memory_bytes`.
- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
- `repositories.memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
0 implies no limit. This value cannot exceed that of the top level `memory_bytes`.
- `repositories.cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU.
This value cannot exceed that of the top level`cgroups_cpu_shares`.
This value cannot exceed that of the top level`cpu_shares`.
For example:
```ruby
# in /etc/gitlab/gitlab.rb
gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
gitaly['cgroups_hierarchy_root'] => "gitaly"
gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb
gitaly['cgroups_cpu_shares'] = 1024
gitaly['cgroups_repositories_count'] => 1000,
gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb
gitaly['cgroups_repositories_cpu_shares'] => 512
gitaly['configuration'] = {
# ...
cgroups: {
mountpoint: '/sys/fs/cgroup',
hierarchy_root: 'gitaly',
memory_bytes: 64424509440, # 60gb
cpu_shares: 1024,
repositories: {
count: 1000,
memory_bytes: 32212254720, # 20gb
cpu_shares: 512,
},
},
}
```
### Configure repository cgroups (legacy method)
@@ -968,10 +1022,16 @@ Make sure to schedule this during off-peak hours and keep the duration short (fo
Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
gitaly['daily_maintenance_start_hour'] = 4
gitaly['daily_maintenance_start_minute'] = 30
gitaly['daily_maintenance_duration'] = '30m'
gitaly['daily_maintenance_storages'] = ["default"]
gitaly['configuration'] = {
# ...
daily_maintenance: {
# ...
start_hour: 4,
start_minute: 30,
duration: '30m',
storages: ['default'],
},
}
```
**For installations from source**
@@ -1018,7 +1078,13 @@ transitioning" mode as follows:
```ruby
# in /etc/gitlab/gitlab.rb
gitaly['auth_transitioning'] = true
gitaly['configuration'] = {
# ...
auth: {
# ...
transitioning: true,
},
}
```
After you have made this change, your [Prometheus query](#verify-authentication-monitoring)
@@ -1038,8 +1104,13 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi
```ruby
# in /etc/gitlab/gitlab.rb
gitaly['auth_token'] = '<new secret token>'
gitaly['configuration'] = {
# ...
auth: {
# ...
token: '<new secret token>',
},
}
```
1. Restart Gitaly:
@@ -1069,7 +1140,13 @@ your Gitaly servers as follows:
```ruby
# in /etc/gitlab/gitlab.rb
gitaly['auth_transitioning'] = false
gitaly['configuration'] = {
# ...
auth: {
# ...
transitioning: false,
},
}
```
WARNING:
@@ -1147,9 +1224,15 @@ setting is discussed in greater detail below.
In `/etc/gitlab/gitlab.rb`, set:
```ruby
gitaly['pack_objects_cache_enabled'] = true
## gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache'
## gitaly['pack_objects_cache_max_age'] = '5m'
gitaly['configuration'] = {
# ...
pack_objects_cache: {
# ...
enabled: true,
# dir: '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache',
# max_age: '5m',
},
}
```
#### `enabled` defaults to `false`
@@ -1315,7 +1398,32 @@ process repositories that do not pass consistency checks.
For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the
following keys (in this example, to disable the `hasDotgit` consistency check):
- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later:
- In [GitLab 15.9](https://gitlab.com/gitlab-org/gitaly/-/issues/4754) and later:
```ruby
ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
gitaly['configuration'] = {
# ...
git: {
# ...
config: [
# Populate a file with one unabbreviated SHA-1 per line.
# See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList
{ key: "fsck.skipList", value: ignored_blobs },
{ key: "fetch.fsck.skipList", value: ignored_blobs },
{ key: "receive.fsck.skipList", value: ignored_blobs },
{ key: "fsck.hasDotgit", value: "ignore" },
{ key: "fetch.fsck.hasDotgit", value: "ignore" },
{ key: "receive.fsck.hasDotgit", value: "ignore" },
{ key: "fsck.missingSpaceBeforeEmail", value: "ignore" },
],
},
}
```
- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) to GitLab 15.8:
```ruby
ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
@@ -1423,7 +1531,13 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
1. Edit `/etc/gitlab/gitlab.rb` and configure `gitaly['gpg_signing_key_path']`:
```ruby
gitaly['gpg_signing_key_path'] = "/etc/gitlab/gitaly/signing_key.gpg"
gitaly['configuration'] = {
# ...
git: {
# ...
signing_key: '/etc/gitlab/gitaly/signing_key.gpg',
},
}
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
Loading