Geo: Secondary site with separate URL redirects to Cloud Native Hybrid primary site
Problem
When setting up a Geo deployment with a Cloud Native Hybrid environment for a primary site and in my testing I've been using an Omnibus secondary, When navigating to the secondary URL you will be redirected to the primary site. This prevents login and general navigation.
In the below screenshot geo-staging-ref-10k
is the primary site and geo-staging-ref-3k
is the secondary. The request is when navigating to the secondary sites sign in page.
Root cause
It appears that this is because the primary site is generating the redirect's Location
header using its own Host
, instead of the secondary site's Host
which the secondary sets in the X-Forwarded-Host
header. It turns out NGINX Ingress controller clobbers X-Forwarded-Host
by default, for security:
If true, NGINX passes the incoming
X-Forwarded-*
headers to upstreams. Use this option when NGINX is behind another L7 proxy / load balancer that is setting these headers.If false, NGINX ignores incoming
X-Forwarded-*
headers, filling them with the request information it sees. Use this option if NGINX is exposed directly to the internet, or it's behind a L3/packet-based load balancer that doesn't alter the source IP in the packets.
Relates to https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/staging-ref/-/issues/49
Workaround
Modify NGINX-Ingress controller chart values.yaml
to include:
controller:
data:
use-forwarded-headers: true
For related context about how to modify configuration, see NGINX Ingress.
GitLab Environment Toolkit
If you are using GitLab Environment Toolkit to deploy a Cloud Native Hybrid environment with Geo, then one way to achieve this NGINX Ingress configuration is by inserting the following into https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/e7a97a0bef10953931a22e365b6222dc7fae6ebb/ansible/roles/gitlab_charts/templates/gitlab.yml.j2#L337 (under the nginx-ingress.controller
key):
{% if (cloud_native_hybrid_geo_role is defined) and (cloud_native_hybrid_geo_role == 'primary') %}
config:
use-forwarded-headers: true
{% endif %}
Or instead of changing GET code, you may add the above as custom configuration specifically for your primary site. There is probably no harm if you set this on secondary sites, except in that they would be configured to be more slightly more open than they absolutely need to be.
And then run:
ansible-playbook -i /path/to/primary/site/ansible/inventory /path/to/playbooks/gitlab_charts.yml
Proposal
- Update Charts to automatically set
use-forwarded-headers: true
when the site is a Geo primary. Possible locations: controller-configmap.yaml, primary.yaml. - Update doc: Charts NGINX Ingress to describe the problem and how to address it in affected versions 15.2 - 15.X.
- Update doc: Charts + Geo to tell people using affected versions to configure this setting.
- Update doc: GitLab 15 specific changes to mention a bug in 15.2 through 15.X when using Geo with separate URLs and Charts.
- Cross-link docs and this issue where appropriate.