Update gitaly and praefect docs with requirement when mixing local / remote gitalies

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

related: gitaly#2183 (closed)

As discovered in gitaly#2140 (comment 247046643), gitaly breaks when the following setup is used:

git_data_dirs({
  "default" => {
          "path" => "/var/opt/gitlab/git-data"
  },
  "second-gitaly" => {
          "gitaly_address" => "tcp://10.168.0.18:8075"
  }
})

The reason is, during an inter gitaly operation, rails will pass the local socket for default as the address to second-gitaly, which then tries to access that socket locally and errors.

We need to document this limitation of gitaly. If a customer wants to mix local and remote gitaly servers, they need to put the IP address of the gitlab box:

git_data_dirs({
  "default" => {
          "gitaly_address" => "tcp://10.24.12.1:8075"
  },
  "second-gitaly" => {
          "gitaly_address" => "tcp://10.168.0.18:8075"
  }
})

This documentation should be in two places. The gitaly installation docs, and the praefect docs