Move Nginx in front of HTTP router rather than behind it

Overview

Currently, the request goes through like this:

---
title: Current
---
graph LR
  subgraph With Nginx
    direction LR
    http-routerN[HTTP router]

    subgraph cellN1[cell1]
      direction LR
      nginx1[nginx1] --> workhorseN1[workhorse1]
      nginx1 --> viteN1[vite1]
      workhorseN1 --> railsN1[rails1]
    end

    subgraph cellN2[cell2]
      direction LR
      nginx2[nginx2] --> workhorseN2[workhorse2]
      nginx2 --> viteN2[vite2]
      workhorseN2 --> railsN2[rails2]
    end
  end

  http-routerN --> nginx1
  http-routerN --> nginx2

  subgraph Without Nginx
    direction LR
    http-router[HTTP router]

    subgraph cell1
      direction LR
      workhorse1 --> rails1
      rails1 --> vite1
    end

    subgraph cell2
      direction LR
      workhorse2 --> rails2
      rails2 --> vite2
    end
  end

  http-router --> workhorse1
  http-router --> workhorse2

The proposal is to move Nginx in front of HTTP router and potentially make Nginx mandatory:

---
title: Proposed
---
graph LR
  nginx[nginx HTTP/2 + HTTPS] --> http-router[HTTP router]

  subgraph cell2
    direction LR
    workhorse2 --> rails2
    vite2
  end

  subgraph cell1
    direction LR
    workhorse1 --> rails1
    vite1
  end

  nginx --> vite1
  nginx --> vite2
  http-router --> workhorse1
  http-router --> workhorse2

It's probably very difficult to support vite in the second cell though. Perhaps we should only support that for the first cell, and the rest should use webpack instead.

---
title: Proposed simplified
---
graph LR
  nginx[nginx HTTP/2 + HTTPS] --> http-router[HTTP router]

  subgraph cell2
    direction LR
    workhorse2 --> rails2
    rails2 --> webpack2
  end

  subgraph cell1
    direction LR
    workhorse1 --> rails1
    vite1
  end

  nginx --> vite1
  http-router --> workhorse1
  http-router --> workhorse2

This addresses or simplifies the following concerns using GDK: (they're not applicable on production)

• SSL/TLS termination with Nginx so HTTP router doesn't need to support HTTPS: #2304 (comment 2244960195)
• CF-Connecting-IP header can be trusted when Nginx can drop it from the client: gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#24 (comment 2313184722)
• We can handle X-Forwarded-For header properly so that we don't need to implement it in HTTP router: gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#24 (closed)
• We can promote HTTP/2 for Vite development experience: gitlab#423851 (comment 2309322804)
• If Vite can perform well on GDK, we don't need to precompile assets on CI: gitlab!172421 (merged)
  • This has been done, and it's not something we need to revisit, but this will provide an option to NOT precompile assets like before, simplifying CI configurations

Impacted categories

The following categories relate to this issue:

• gdk-reliability - e.g. When a GDK action fails to complete.
• gdk-usability - e.g. Improvements or suggestions around how the GDK functions.
• gdk-performance - e.g. When a GDK action is slow or times out.

Steps to replicate (optional)

Proposal (optional)

Environment (optional)

• Operating system name: <!-- output of `uname -a` command -->
• Architecture: <!-- output of `arch` command -->
• The contents of your gdk.yml (if any)
• Ruby version: <!-- output of `ruby --version` command -->
• GDK version: <!-- output of `git rev-parse --short HEAD` command -->