Commit 0923fbf7 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis Committed by Evan Read
Browse files

Clean up Gitaly admin docs 

- Add missing path to Gitaly configuration
- Properly list steps for Gitaly configurations
- Move sections around
parent c61c3b68

doc/administration/gitaly/index.md

+359 −311
Original line number Diff line number Diff line
@@ -2,45 +2,35 @@ 


[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that

provides high-level RPC access to Git repositories. Without it, no other

components can read or write Git data.

components can read or write Git data. GitLab components that access Git

repositories (gitlab-rails, gitlab-shell, gitlab-workhorse, etc.) act as clients

to Gitaly. End users do not have direct access to Gitaly.



GitLab components that access Git repositories (gitlab-rails,

gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do

not have direct access to Gitaly.

In the rest of this page, Gitaly server is referred to the standalone node that

only runs Gitaly, and Gitaly client to the GitLab Rails node that runs all other

processes except Gitaly.



## Configuring Gitaly



The Gitaly service itself is configured via a TOML configuration file.

This file is documented [in the gitaly

This file is documented [in the Gitaly

repository](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md).



To change a Gitaly setting in Omnibus you can use

`gitaly['my_setting']` in `/etc/gitlab/gitlab.rb`. Changes will be applied

when you run `gitlab-ctl reconfigure`.

In case you want to change some of its settings:



```ruby

gitaly['prometheus_listen_addr'] = 'localhost:9236'

```

**For Omnibus GitLab**



To change a Gitaly setting in installations from source you can edit

`/home/git/gitaly/config.toml`. Changes will be applied when you run

`service gitlab restart`.



```toml

prometheus_listen_addr = "localhost:9236"

```

1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676).

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).



## Client-side GRPC logs

**For installations from source**



Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC

client has its own log file which may contain useful information when

you are seeing Gitaly errors. You can control the log level of the

gRPC client with the `GRPC_LOG_LEVEL` environment variable. The

default level is `WARN`.

1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example).

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).



## Running Gitaly on its own server



> This is an optional way to deploy Gitaly which can benefit GitLab

This is an optional way to deploy Gitaly which can benefit GitLab

installations that are larger than a single machine. Most

installations will be better served with the default configuration

used by Omnibus and the GitLab source installation guide.
@@ -53,76 +43,78 @@ But since 11.8 the indexer uses Gitaly for data access as well. NFS can still 
be leveraged for redudancy on block level of the Git data. But only has to

be mounted on the Gitaly server.



NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend

using EFS as it may impact GitLab's performance. Please review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs)

NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended

to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs)

for more details.



### Network architecture



-   gitlab-rails shards repositories into "repository storages"

-   `gitlab-rails/config/gitlab.yml` contains a map from storage names to

    (Gitaly address, Gitaly token) pairs

The following list depicts what the network architecture of Gitaly is:



- GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md).

- `/config/gitlab.yml` contains a map from storage names to

  `(Gitaly address, Gitaly token)` pairs.

- the `storage name` -\> `(Gitaly address, Gitaly token)` map in

    `gitlab.yml` is the single source of truth for the Gitaly network

    topology

-   a (Gitaly address, Gitaly token) corresponds to a Gitaly server

-   a Gitaly server hosts one or more storages

  `/config/gitlab.yml` is the single source of truth for the Gitaly network

  topology.

- A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server.

- A Gitaly server hosts one or more storages.

- Gitaly addresses must be specified in such a way that they resolve

    correctly for ALL Gitaly clients

-   Gitaly clients are: unicorn, sidekiq, gitlab-workhorse,

    gitlab-shell, Elasticsearch Indexer, and Gitaly itself

-   special case: a Gitaly server must be able to make RPC calls **to

    itself** via its own (Gitaly address, Gitaly token) pair as

    specified in `gitlab-rails/config/gitlab.yml`

-   Gitaly servers must not be exposed to the public internet



Gitaly network traffic is unencrypted by default, but supports

[TLS](#tls-support). Authentication is done through a static token.



NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to

restrict access to your Gitaly server.

  correctly for ALL Gitaly clients.

- Gitaly clients are: Unicorn, Sidekiq, gitlab-workhorse,

  gitlab-shell, Elasticsearch Indexer, and Gitaly itself.

- A Gitaly server must be able to make RPC calls **to itself** via its own

  `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`.

- Gitaly servers must not be exposed to the public internet as Gitaly's network

  traffic is unencrypted by default. The use of firewall is highly recommended

  to restrict access to the Gitaly server. Another option is to

  [use TLS](#tls-support).

- Authentication is done through a static token which is shared among the Gitaly

  and GitLab Rails nodes.



Below we describe how to configure a Gitaly server at address

`gitaly.internal:8075` with secret token `abc123secret`. We assume

your GitLab installation has two repository storages, `default` and

`storage1`.



### Installation

### 1. Installation



First install Gitaly using either Omnibus or from source.

First install Gitaly using either Omnibus GitLab or install it from source:



Omnibus: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab

- For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab

  package you want using **steps 1 and 2** from the GitLab downloads page but

  **_do not_** provide the `EXTERNAL_URL=` value.

- From source: [Install Gitaly](../../install/installation.md#install-gitaly).



Source: [Install Gitaly](../../install/installation.md#install-gitaly)

### 2. Client side token configuration



### Client side token configuration

Configure a token on the instance that runs the GitLab Rails application.



Configure a token on the client side.

**For Omnibus GitLab**



Omnibus installations:

1. On the client node(s), edit `/etc/gitlab/gitlab.rb`:



   ```ruby

# /etc/gitlab/gitlab.rb

   gitlab_rails['gitaly_token'] = 'abc123secret'

   ```



Source installations:

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).



**For installations from source**



1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`:



   ```yaml

# /home/git/gitlab/config/gitlab.yml

   gitlab:

     gitaly:

       token: 'abc123secret'

   ```



You need to reconfigure (Omnibus) or restart (source) for these

changes to be picked up.

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).



### Gitaly server configuration

### 3. Gitaly server configuration



Next, on the Gitaly server, we need to configure storage paths, enable

Next, on the Gitaly server, you need to configure storage paths, enable

the network listener and configure the token.



NOTE: **Note:** if you want to reduce the risk of downtime when you enable
@@ -137,11 +129,14 @@ the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gi 
from an existing GitLab server to the Gitaly server. Without this shared secret,

Git operations in GitLab will result in an API error.



NOTE: **Note:** In most or all cases the storage paths below end in `/repositories` which is

different than `path` in `git_data_dirs` of Omnibus installations. Check the

directory layout on your Gitaly server to be sure.

NOTE: **Note:**

In most or all cases, the storage paths below end in `/repositories` which is

not that case with `path` in `git_data_dirs` of Omnibus GitLab installations.

Check the directory layout on your Gitaly server to be sure.



Omnibus installations:

**For Omnibus GitLab**



1. Edit `/etc/gitlab/gitlab.rb`:



   <!--

   updates to following example must also be made at
@@ -176,8 +171,8 @@ gitaly['listen_addr'] = "0.0.0.0:8075" 
   gitaly['auth_token'] = 'abc123secret'



   gitaly['storage'] = [

  { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' },

  { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' },

     { 'name' => 'default' },

     { 'name' => 'storage1' },

   ]



   # To use TLS for Gitaly you need to add
@@ -186,10 +181,17 @@ gitaly['certificate_path'] = "path/to/cert.pem" 
   gitaly['key_path'] = "path/to/key.pem"

   ```



Source installations:

   NOTE: **Note:**

   In some cases, you'll have to set `path` for `gitaly['storage']` in the

   format `'path' => '/mnt/gitlab/<storage name>/repositories'`.



1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).



**For installations from source**



1. On the client node(s), edit `/home/git/gitaly/config.toml`:



   ```toml

# /home/git/gitaly/config.toml

   listen_addr = '0.0.0.0:8075'

   tls_listen_addr = '0.0.0.0:9999'
@@ -202,23 +204,24 @@ token = 'abc123secret' 


   [[storage]]

   name = 'default'

path = '/mnt/gitlab/default/repositories'



   [[storage]]

   name = 'storage1'

path = '/mnt/gitlab/storage1/repositories'

   ```



Again, reconfigure (Omnibus) or restart (source).

   NOTE: **Note:**

   In some cases, you'll have to set `path` for each `[[storage]]` in the

   format `path = '/mnt/gitlab/<storage name>/repositories'`.



### Converting clients to use the Gitaly server

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).



Now as the final step update the client machines to switch from using

their local Gitaly service to the new Gitaly server you just

configured. This is a risky step because if there is any sort of

network, firewall, or name resolution problem preventing your GitLab

server from reaching the Gitaly server then all Gitaly requests will

fail.

### 4. Converting clients to use the Gitaly server



As the final step, you need to update the client machines to switch from using

their local Gitaly service to the new Gitaly server you just configured. This

is a risky step because if there is any sort of network, firewall, or name

resolution problem preventing your GitLab server from reaching the Gitaly server,

then all Gitaly requests will fail.



Additionally, you need to

[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab).
@@ -227,10 +230,11 @@ We assume that your Gitaly server can be reached at 
`gitaly.internal:8075` from your GitLab server, and that Gitaly can read and

write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively.



Omnibus installations:

**For Omnibus GitLab**



1. Edit `/etc/gitlab/gitlab.rb`:



   ```ruby

# /etc/gitlab/gitlab.rb

   git_data_dirs({

     'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' },

     'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' },
@@ -239,29 +243,80 @@ git_data_dirs({ 
   gitlab_rails['gitaly_token'] = 'abc123secret'

   ```



Source installations:

   NOTE: **Note:**

   In some cases, you'll have to set `path` for each `git_data_dirs` in the

   format `'path' => '/mnt/gitlab/<storage name>'`.



1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).

1. Tail the logs to see the requests:



   ```sh

   sudo gitlab-ctl tail gitaly

   ```



**For installations from source**



1. Edit `/home/git/gitlab/config/gitlab.yml`:



   ```yaml

# /home/git/gitlab/config/gitlab.yml

   gitlab:

     repositories:

       storages:

         default:

        path: /mnt/gitlab/default/repositories

           gitaly_address: tcp://gitaly.internal:8075

         storage1:

        path: /mnt/gitlab/storage1/repositories

           gitaly_address: tcp://gitaly.internal:8075



     gitaly:

       token: 'abc123secret'

   ```



Now reconfigure (Omnibus) or restart (source). When you tail the

Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or

`tail -f /home/git/gitlab/log/gitaly.log`) you should see requests

coming in. One sure way to trigger a Gitaly request is to clone a

repository from your GitLab server over HTTP.

   NOTE: **Note:**

   In some cases, you'll have to set `path` for each of the `storages` in the

   format `path: /mnt/gitlab/<storage name>/repositories`.



1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).

1. Tail the logs to see the requests:



   ```sh

   tail -f /home/git/gitlab/log/gitaly.log

   ```



When you tail the Gitaly logs on your Gitaly server you should see requests

coming in. One sure way to trigger a Gitaly request is to clone a repository

from your GitLab server over HTTP.



### Disabling the Gitaly service in a cluster environment



If you are running Gitaly [as a remote

service](#running-gitaly-on-its-own-server) you may want to disable

the local Gitaly service that runs on your GitLab server by default.

Disabling Gitaly only makes sense when you run GitLab in a custom

cluster configuration, where different services run on different

machines. Disabling Gitaly on all machines in the cluster is not a

valid configuration.



To disable Gitaly on a client node:



**For Omnibus GitLab**



1. Edit `/etc/gitlab/gitlab.rb`:



   ```ruby

   gitaly['enable'] = false

   ```



1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).



**For installations from source**



1. Edit `/etc/default/gitlab`:



   ```shell

   gitaly_enabled=false

   ```



1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).



## TLS support
@@ -271,29 +326,24 @@ Gitaly supports TLS encryption. To be able to communicate 
with a Gitaly instance that listens for secure connections you will need to use `tls://` url

scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration.



The admin needs to bring their own certificate as we do not provide that automatically.

The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).

You will need to bring your own certificates as this isn't provided automatically.

The certificate to be used needs to be installed on all Gitaly nodes and on all

client nodes that communicate with it following the procedure described in

[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).



Note that it is possible to configure Gitaly servers with both an

NOTE: **Note:**

It is possible to configure Gitaly servers with both an

unencrypted listening address `listen_addr` and an encrypted listening

address `tls_listen_addr` at the same time. This allows you to do a

gradual transition from unencrypted to encrypted traffic, if necessary.



To observe what type of connections are actually being used in a

production environment you can use the following Prometheus query:



```

sum(rate(gitaly_connections_total[5m])) by (type)

```



### Example TLS configuration

To configure Gitaly with TLS:



### Omnibus installations:

**For Omnibus GitLab**



#### On client nodes:

1. On the client nodes, edit `/etc/gitlab/gitlab.rb`:



   ```ruby

# /etc/gitlab/gitlab.rb

   git_data_dirs({

     'default' => { 'gitaly_address' => 'tls://gitaly.internal:9999' },

     'storage1' => { 'gitaly_address' => 'tls://gitaly.internal:9999' },
@@ -302,7 +352,8 @@ git_data_dirs({ 
   gitlab_rails['gitaly_token'] = 'abc123secret'

   ```



#### On Gitaly server nodes:

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).

1. On the Gitaly server nodes, edit `/etc/gitlab/gitlab.rb`:



   ```ruby

   gitaly['tls_listen_addr'] = "0.0.0.0:9999"
@@ -310,30 +361,33 @@ gitaly['certificate_path'] = "path/to/cert.pem" 
   gitaly['key_path'] = "path/to/key.pem"

  ```



### Source installations:

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).



**For installations from source**



#### On client nodes:

1. On the client nodes, edit `/home/git/gitlab/config/gitlab.yml`:



   ```yaml

# /home/git/gitlab/config/gitlab.yml

   gitlab:

     repositories:

       storages:

         default:

        path: /mnt/gitlab/default/repositories

           gitaly_address: tls://gitaly.internal:9999

         storage1:

        path: /mnt/gitlab/storage1/repositories

           gitaly_address: tls://gitaly.internal:9999



     gitaly:

       token: 'abc123secret'

   ```



#### On Gitaly server nodes:

   NOTE: **Note:**

   In some cases, you'll have to set `path` for each of the `storages` in the

   format `path: /mnt/gitlab/<storage name>/repositories`.



1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).

1. On the Gitaly server nodes, edit `/home/git/gitaly/config.toml`:



   ```toml

# /home/git/gitaly/config.toml

   tls_listen_addr = '0.0.0.0:9999'



   [tls]
@@ -341,98 +395,59 @@ certificate_path = '/path/to/cert.pem' 
   key_path = '/path/to/key.pem'

   ```



## Gitaly-ruby

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).



To observe what type of connections are actually being used in a

production environment you can use the following Prometheus query:



```

sum(rate(gitaly_connections_total[5m])) by (type)

```



## `gitaly-ruby`



Gitaly was developed to replace Ruby application code in gitlab-ce/ee.

Gitaly was developed to replace the Ruby application code in GitLab.

In order to save time and/or avoid the risk of rewriting existing

application logic, in some cases we chose to copy some application code

from gitlab-ce into Gitaly almost as-is. To be able to run that code, we

made gitaly-ruby, which is a sidecar process for the main Gitaly Go

process. Some examples of things that are implemented in gitaly-ruby are

RPC's that deal with wiki's, and RPC's that create commits on behalf of

from GitLab into Gitaly almost as-is. To be able to run that code,

`gitaly-ruby` was created, which is a "sidecar" process for the main Gitaly Go

process. Some examples of things that are implemented in `gitaly-ruby` are

RPCs that deal with wikis, and RPCs that create commits on behalf of

a user, such as merge commits.



### Number of gitaly-ruby workers

### Number of `gitaly-ruby` workers



Gitaly-ruby has much less capacity than Gitaly itself. If your Gitaly

server has to handle a lot of request, the default setting of having

just 1 active gitaly-ruby sidecar might not be enough. If you see

ResourceExhausted errors from Gitaly it's very likely that you have not

enough gitaly-ruby capacity.

`gitaly-ruby` has much less capacity than Gitaly itself. If your Gitaly

server has to handle a lot of requests, the default setting of having

just one active `gitaly-ruby` sidecar might not be enough. If you see

`ResourceExhausted` errors from Gitaly, it's very likely that you have not

enough `gitaly-ruby` capacity.



You can increase the number of gitaly-ruby processes on your Gitaly

You can increase the number of `gitaly-ruby` processes on your Gitaly

server with the following settings.



Omnibus:

**For Omnibus GitLab**



1. Edit `/etc/gitlab/gitlab.rb`:



   ```ruby

# /etc/gitlab/gitlab.rb

   # Default is 2 workers. The minimum is 2; 1 worker is always reserved as

   # a passive stand-by.

   gitaly['ruby_num_workers'] = 4

   ```



Source:

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).



**For installations from source**



1. Edit `/home/git/gitaly/config.toml`:



   ```toml

# /home/git/gitaly/config.toml

   [gitaly-ruby]

   num_workers = 4

   ```



### Observing gitaly-ruby traffic



Gitaly-ruby is a somewhat hidden, internal implementation detail of

Gitaly. There is not that much visibility into what goes on inside

gitaly-ruby processes.



If you have Prometheus set up to scrape your Gitaly process, you can see

request rates and error codes for individual RPC's in gitaly-ruby by

querying `grpc_client_handled_total`. Strictly speaking this metric does

not differentiate between gitaly-ruby and other RPC's, but in practice

(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal

calls from the main Gitaly process to one of its gitaly-ruby sidecars.



Assuming your `grpc_client_handled_total` counter only observes Gitaly,

the following query shows you RPC's are (most likely) internally

implemented as calls to gitaly-ruby.



```

sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0

```



## Disabling or enabling the Gitaly service in a cluster environment



If you are running Gitaly [as a remote

service](#running-gitaly-on-its-own-server) you may want to disable

the local Gitaly service that runs on your GitLab server by default.



> 'Disabling Gitaly' only makes sense when you run GitLab in a custom

cluster configuration, where different services run on different

machines. Disabling Gitaly on all machines in the cluster is not a

valid configuration.



If you are setting up a GitLab cluster where Gitaly does not need to

run on all machines, you can disable the Gitaly service in your

Omnibus installation, add the following line to `/etc/gitlab/gitlab.rb`:



```ruby

gitaly['enable'] = false

```



When you run `gitlab-ctl reconfigure` the Gitaly service will be

disabled.



To disable the Gitaly service in a GitLab cluster where you installed

GitLab from source, add the following to `/etc/default/gitlab` on the

machine where you want to disable Gitaly.



```shell

gitaly_enabled=false

```



When you run `service gitlab restart` Gitaly will be disabled on this

particular machine.

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).



## Eliminating NFS altogether
@@ -448,11 +463,15 @@ a few things that you need to do: 
1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).

1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only).



NOTE: **Note:** One current feature of GitLab still requires a shared directory (NFS): [GitLab Pages](../../user/project/pages/index.md).

NOTE: **Note:**

One current feature of GitLab that still requires a shared directory (NFS) is

[GitLab Pages](../../user/project/pages/index.md).

There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196)

to eliminate the need for NFS to support GitLab Pages.



## Troubleshooting Gitaly in production

## Troubleshooting



### `gitaly-debug`



Since GitLab 11.6, Gitaly comes with a command-line tool called

`gitaly-debug` that can be run on a Gitaly server to aid in
@@ -462,3 +481,32 @@ Git clone speed for a specific repository on the server. 


For an up to date list of sub-commands see [the gitaly-debug

README](https://gitlab.com/gitlab-org/gitaly/blob/master/cmd/gitaly-debug/README.md).



### Client side GRPC logs



Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC

client has its own log file which may contain useful information when

you are seeing Gitaly errors. You can control the log level of the

gRPC client with the `GRPC_LOG_LEVEL` environment variable. The

default level is `WARN`.



### Observing `gitaly-ruby` traffic



[`gitaly-ruby`](#gitaly-ruby) is an internal implementation detail of Gitaly,

so, there's not that much visibility into what goes on inside

`gitaly-ruby` processes.



If you have Prometheus set up to scrape your Gitaly process, you can see

request rates and error codes for individual RPCs in `gitaly-ruby` by

querying `grpc_client_handled_total`. Strictly speaking, this metric does

not differentiate between `gitaly-ruby` and other RPCs, but in practice

(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal

calls from the main Gitaly process to one of its `gitaly-ruby` sidecars.



Assuming your `grpc_client_handled_total` counter only observes Gitaly,

the following query shows you RPCs are (most likely) internally

implemented as calls to `gitaly-ruby`:



```

sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0

```