Skip to content

Gitaly unreachable due to grpc not correctly using no_proxy environment variable in helm based install in GitLab 17.10.0

Summary

While updating Gitlab (Helm based install) from 17.9.2 to Gitlab 17.10.0 we encountered issues with our GRPC connection to our external Gitaly server. We are using proxy settings to connect to different services, but the connection to the gitaly server is included as a hostname in the no_proxy extraEnv variable. This does not seem to be working after upgrading to 17.10.0. We can see most connections to gitaly working just fine, but some connections fail. Examples are opening files in the web IDE, pulling a repo to your local machine, running pipelines on agents that fetch the repo and probably more. Our workaround is to add the server IP address to the no_proxy variable, which seems to be working fine for now, but is not a desirable long-term solution. We suspect the underlying issue is with the older grpc versions, see https://github.com/grpc/grpc-go/issues/7556

Steps to reproduce

  • Set up a gitlab helm release

  • Set up a seperate gitaly server

  • Set up a proxy settings that cannot connect to the gitaly server

  • Set the proxy variables for the gitlab helm release, adding the gitaly hostname to the no_proxy variable

  • Test the connection by actually pulling a repo, running a pipeline that pulls the repo e.t.c.

  • Test the workaround by adding the ip of the gitaly server instead of the hostname

What is the current bug behavior?

We suspect the actual issue is caused by an older grpc package (1.67.1) of which you may be using the new client that resolves the hostname on the client instead of against the proxy. This means it will check for the ip in the no_proxy variable instead of the hostname, doesnt find it and connects over ip through the proxy.

What is the expected correct behavior?

Using a newer grpc package (1.71.0+) should no longer resolve the hostname before using the proxy, causing the connection to correctly skip the proxy based on the no_proxy settings.

Results of GitLab environment info

Expand for output related to GitLab environment info 

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.6.6
Bundler Version:2.6.6
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.2.4
Go Version:     unknown

GitLab information
Version:        17.10.0
Revision:       d8c1ba94b65
Directory:      /srv/gitlab
DB Adapter:     PostgreSQL
DB Version:     16.6
URL:            redacted
HTTP Clone URL: redacted
SSH Clone URL:  redacted
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml

GitLab Shell
Version:        14.41.0
Repository storages:
- default:      tcp://redacted:8075
GitLab Shell path:              /home/git/gitlab-shell

Gitaly
- default Address:      tcp://redacted:8075
- default Version:      17.10.0
- default Git Version:  2.48.1.gl1

Results of GitLab application Check

all application checks seem ok

Possible fixes

Upgrade to grpc 1.71.0 or later