Geo: Secondary site with separate URL redirects to Cloud Native Hybrid primary site

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem

When setting up a Geo deployment with a Cloud Native Hybrid environment for a primary site and in my testing I've been using an Omnibus secondary, When navigating to the secondary URL you will be redirected to the primary site. This prevents login and general navigation.

In the below screenshot geo-staging-ref-10k is the primary site and geo-staging-ref-3k is the secondary. The request is when navigating to the secondary sites sign in page.

Screenshot_2022-11-04_at_14.17.14

Root cause

It appears that this is because the primary site is generating the redirect's Location header using its own Host, instead of the secondary site's Host which the secondary sets in the X-Forwarded-Host header. It turns out NGINX Ingress controller clobbers X-Forwarded-Host by default, for security:

If true, NGINX passes the incoming X-Forwarded-* headers to upstreams. Use this option when NGINX is behind another L7 proxy / load balancer that is setting these headers.

If false, NGINX ignores incoming X-Forwarded-* headers, filling them with the request information it sees. Use this option if NGINX is exposed directly to the internet, or it's behind a L3/packet-based load balancer that doesn't alter the source IP in the packets.

Relates to https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/issues/49

Workaround

Modify NGINX-Ingress controller chart values.yaml to include:

controller:
  data:
    use-forwarded-headers: true

For related context about how to modify configuration, see NGINX Ingress.

GitLab Environment Toolkit

If you are using GitLab Environment Toolkit to deploy a Cloud Native Hybrid environment with Geo, then one way to achieve this NGINX Ingress configuration is by inserting the following into https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/e7a97a0bef10953931a22e365b6222dc7fae6ebb/ansible/roles/gitlab_charts/templates/gitlab.yml.j2#L337 (under the nginx-ingress.controller key):

{% if (cloud_native_hybrid_geo_role is defined) and (cloud_native_hybrid_geo_role == 'primary') %}
    config:
      use-forwarded-headers: true
{% endif %}

Or instead of changing GET code, you may add the above as custom configuration specifically for your primary site. There is probably no harm if you set this on secondary sites, except in that they would be configured to be more slightly more open than they absolutely need to be.

And then run:

ansible-playbook -i /path/to/primary/site/ansible/inventory /path/to/playbooks/gitlab_charts.yml

Proposal

  • Update Charts to automatically set use-forwarded-headers: true when the site is a Geo primary. Possible locations: controller-configmap.yaml, primary.yaml.
  • Update doc: Charts NGINX Ingress to describe the problem and how to address it in affected versions 15.2 - 15.X.
  • Update doc: Charts + Geo to tell people using affected versions to configure this setting.
  • Update doc: GitLab 15 specific changes to mention a bug in 15.2 through 15.X when using Geo with separate URLs and Charts.
  • Cross-link docs and this issue where appropriate.
Edited by 🤖 GitLab Bot 🤖