Geo: Proxy request buffering must be disabled for primary site's Internal URL (extra ingress)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

The original bug is blocked by the inability to disable proxy request buffering for specific paths in Ingress NGINX in a primary site deployed by the GitLab Helm chart. Proxy request buffering causes requests from the secondary site's GitLab-Shell to the primary site's Workhorse to hang.

Apparently proxy request buffering protects against Slowloris attacks (initiating many HTTP connections and holding them open).

It seems that primary Geo sites deployed by the Helm chart already require a second Ingress. That so-called "extra" Ingress already handles requests from the secondary site to the primary site's "Internal URL". This Ingress is not exposed to users, so it should be safe to disable all proxy request buffering using nginx.ingress.kubernetes.io/proxy-request-buffering: "off".

The secondary site is unaffected because it doesn't receive requests from a GitLab-Shell in another site.

In this issue we need to do the following:

  1. We need to validate this proposed solution works (deploy Geo with Helm chart, enable the Git over SSH feature flags, and test Git over SSH)
  2. Then document it (need the NGINX annotation on the primary site's "Extra" Ingress, enable the Git over SSH feature flags).

Steps to reproduce

Example Project

What is the current bug behavior?

This manifests in Geo when:

  1. proxy request buffering is enabled
  2. geo_proxy_fetch_ssh_to_primary and/or geo_proxy_push_ssh_to_primary is enabled
  3. a Git over SSH request from the secondary is proxied

See related issue: #454707 for more logs and context on the original problem which the changes behind the feature flags were intended to solve, if not for proxy request buffering causing a different problem.

What is the expected correct behavior?

Proxy request buffering breaks the new Git over SSH logic currently behind geo_proxy_fetch_ssh_to_primary and geo_proxy_push_ssh_to_primary.

It is already required to make the primary site receive requests from two Ingresses. So you can disable proxy request buffering on the primary site's "extra" ingress (the one that only handles "internal" requests from secondary sites).

The webservice chart has a extraIngress for this purpose. You can set custom annotations for this Ingress only:

gitlab:
  webservice:
    extraIngress:
      enabled: true
      hostname: gitlab.london.example.com
      annotations:
         nginx.ingress.kubernetes.io/proxy-request-buffering: "off"

Proposal:

Copied from Mike's comment below to highlight here

  1. Try to validate: if we successfully use Git over SSH against a secondary site by setting this annotation (and enabling the FFs), then the only thing that needs to happen for Geo is:
    1. Documentation needs to say to set the annotation 3. Geo just needs the new Git over SSH codepaths to be enabled by default. But since the FF doesn't make Git over SSH work for Cells for GitLab.com, then we may not want to enable this codepath for all cases. If that's the case, then let's just enable the new Git over SSH codepaths for all environments except GitLab.com. Then we can close #454707.
    2. Cells will still need to figure out how to make Git over SSH proxying work, but it looks very likely to me that they can resolve the problem in the same way, by using an "Internal/Extra" Ingress to handle "internal" requests, so it'd be safe from outside attacks. In that case, they can then remove the conditional for GitLab.com.
    3. Another shortcut to consider: If SSH proxying in Cells doesn't currently work, then enabling the FFs by default, or removing them, has no adverse impact to Cells. Not working => still not working does not have an impact. So we don't need to bother with a GitLab.com conditional.
Edited by 🤖 GitLab Bot 🤖