Skip to content

Gitlab Gitaly version 6.4.1 Port Name Prefix Changes Break Istio Injected Installations

Summary

Recent template changes to the Gitaly chart which add either a prefix of grpc- or tls- to the service port name break how a Gitlab installation which is Istio-injected handle the traffic.

Steps to reproduce

Install Gitlab 6.4.1 with gitaly.tls disabled and istio-injection=enabled set on namespace. Attempt to clone a repository from installation:

❯ git clone https://gitlab.example.com/gitlab-instance-e85c0ebc/woohooo.git
Cloning into 'woohooo'...
Username for 'https://gitlab.example.com': root
Password for 'https://root@gitlab.example.com':
remote: The git server, Gitaly, is not available at this time. Please contact your administrator.
fatal: unable to access 'https://gitlab.example.com/gitlab-instance-e85c0ebc/woohooo.git/': The requested URL returned error: 503

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

global:
  # added to help with Gitlab sub-chart configuration
  image:
    pullPolicy: IfNotPresent
  hosts:
    domain: example.com
    gitlab:
      name: gitlab.example.com
    registry:
      name: registry.example.com
  rails:
    bootstrap:
      enabled: false
gitlab-runner:
  resources:
    requests:
      cpu: 10m
    limits: {}
gitlab:
  webservice:
    minReplicas: 1
    maxReplicas: 1
    helmTests:
      enabled: false
  sidekiq:
    minReplicas: 1
    maxReplicas: 1
  gitlab-shell:
    minReplicas: 1
    maxReplicas: 1
  gitaly:
    persistence:
      size: 256Mi
    resources:
      ## values raised to help pass CI after default values for gitaly are fixed then can revert to original request.
      #requests:
      #  cpu: 50m
      #limits: {}
      requests:
        cpu: 400m
        memory: 600Mi
      limits:
        cpu: 400m
        memory: 600Mi
  shared-secrets:
    resources:
      requests:
        cpu: 10m
      limits: {}
  migrations:
    resources:
      requests:
        cpu: 10m
      limits: {}
  toolbox:
    persistence:
      size: 256Mi
    resources:
      requests:
        cpu: 10m
      limits: {}
registry:
  hpa:
    minReplicas: 1
    maxReplicas: 1
postgresql:
  persistence:
    size: 256Mi
  metrics:
    resources:
      requests:
        cpu: 10m
      limits: {}
minio:
  persistence:
    size: 256Mi
  resources:
    requests:
      cpu: 50m
    limits: {}
redis:
  master:
    persistence:
      size: 256Mi
  slave:
    persistence:
      size: 256Mi
            #limits: {}
            requests:
              cpu: 400m
              memory: 600Mi
            limits:
              cpu: 400m
              memory: 600Mi
        shared-secrets:
          resources:
            requests:
              cpu: 10m
            limits: {}
        migrations:
          resources:
            requests:
              cpu: 10m
            limits: {}
        toolbox:
          persistence:
            size: 256Mi
          resources:
            requests:
              cpu: 10m
            limits: {}
      registry:
        hpa:
          minReplicas: 1
          maxReplicas: 1
      postgresql:
        persistence:
          size: 256Mi
        metrics:
          resources:
            requests:
              cpu: 10m
            limits: {}
      minio:
        persistence:
          size: 256Mi
        resources:
          requests:
            cpu: 50m
          limits: {}
      redis:
        master:
          persistence:
            size: 256Mi
        slave:
          persistence:
            size: 256Mi

Current behavior

Command exits unsuccessfully says gitaly returned a 503

Expected behavior

Successful clone of repository

Versions

  • Chart: 6.4.1
  • Platform:
    • Cloud: AWS
    • Self-hosted: RKE2
  • Kubernetes: (kubectl version)
    • Client: v1.23.4
    • Server: v1.24.4
  • Helm: (helm version)
    • Client: v3.8.0
    • Server: v0.34.0 (fluxv2)

Relevant logs

Webservice workhorse container log:

{"level":"error","msg":"2022/10/06 19:05:07 [ERR] yamux: Invalid protocol version: 72","time":"2022-10-06T19:05:07Z"}
{"correlation_id":"01GEQAZT1JX7S7AVFW78YY4GPP","error":"handleGetInfoRefs: rpc error: code = Unavailable desc = connection closed before server preface received","level":"error","method":"GET","msg":"","time":"2022-10-06T19:05:07Z","uri":"/test/test1.git/info/refs?service=git-upload-pack"}
{"content_type":"text/plain; charset=utf-8","correlation_id":"01GEQAZT1JX7S7AVFW78YY4GPP","duration_ms":373,"host":"gitlab.bigbang.dev","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.6:34845","remote_ip":"127.0.0.6","route":"^/.+\\.git/info/refs\\z","status":503,"system":"http","time":"2022-10-06T19:05:07Z","ttfb_ms":373,"uri":"/test/test1.git/info/refs?service=git-upload-pack","user_agent":"git/2.34.1","written_bytes":90}
{"correlation_id":"01GEQAZTTNX2RD8ZBCB98B7FJ5","error":"handleGetInfoRefs: rpc error: code = Unavailable desc = connection closed before server preface received","level":"error","method":"GET","msg":"","time":"2022-10-06T19:05:08Z","uri":"/test/test1.git/info/refs?service=git-upload-pack"}

webservice istio-proxy container logs:

[2022-10-06T18:41:14.512Z] "GET /gitlab-instance-e85c0ebc/woohooo.git/info/refs?service=git-upload-pack HTTP/1.1" 503 - via_upstream - "-" 0 90 405 404 "10.42.0.0" "git/2.37.0" "0bc3ffd1-abab-9fb6-8c9b-986b7ea05e85" "gitlab.bigbang.dev" "10
.42.3.21:8181" inbound|8181|| 127.0.0.6:55661 10.42.3.21:8181 10.42.0.0:0 outbound_.8181_._.gitlab-webservice-default.gitlab.svc.cluster.local default traceID=8282565a0450a751cbfbe735d95cb93f
[2022-10-06T18:41:15.288Z] "GET /gitlab-instance-e85c0ebc/woohooo.git/info/refs?service=git-upload-pack HTTP/1.1" 503 - via_upstream - "-" 0 90 246 246 "10.42.0.0" "git/2.37.0" "0bc3ffd1-abab-9fb6-8c9b-986b7ea05e85" "gitlab.bigbang.dev" "10
.42.3.21:8181" inbound|8181|| 127.0.0.6:55661 10.42.3.21:8181 10.42.0.0:0 outbound_.8181_._.gitlab-webservice-default.gitlab.svc.cluster.local default traceID=8282565a0450a751cbfbe735d95cb93f
[2022-10-06T18:41:15.585Z] "GET /gitlab-instance-e85c0ebc/woohooo.git/info/refs?service=git-upload-pack HTTP/1.1" 503 - via_upstream - "-" 0 90 426 426 "10.42.0.0" "git/2.37.0" "0bc3ffd1-abab-9fb6-8c9b-986b7ea05e85" "gitlab.bigbang.dev" "10
.42.3.21:8181" inbound|8181|| 127.0.0.6:41249 10.42.3.21:8181 10.42.0.0:0 outbound_.8181_._.gitlab-webservice-default.gitlab.svc.cluster.local default traceID=8282565a0450a751cbfbe735d95cb93f