Geo: Add Geo-secondary-specific routing in Workhorse

What does this MR do?

This MR is initially intended for showing something working end-to-end. Smaller MRs will be broken out for review and merge.

After the child MRs have been merged, this MR can retarget master. And this MR will just make Workhorse use Geo-secondary-specific routing if geo_proxy_url is present in the /api/v4/geo/proxy response.

Where does this fit in the epic?

Description	Issue	MR(s)
Add endpoint /api/v4/geo/proxy	#324664 (closed)	!63343 (merged)
Determine if this is a Geo Proxy by calling /api/v4/geo/proxy	#329670 (closed)	!60769 (merged)
If this is a Geo Proxy, then use the Geo Proxy routing table	#329672 (closed)	!63321 (merged) You are here
Cache the result of the request	#329671 (closed)	!66491 (merged)
[Feature flag] Rollout of geo_secondary_proxy and GEO_SECONDARY_PROXY	#325732 (closed)

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

• I have included a changelog entry, or it's not needed. (Does this MR need a changelog?) => Not needed. The API endpoint is feature flagged, and the major workhorse changes are behind environment variable as well.
• I have added/updated documentation, or it's not needed. (Is documentation required?)
• I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
• I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
• I have self-reviewed this MR per code review guidelines.
• This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
• I have followed the style guides.
• This change is backwards compatible across updates, or this does not apply.

Availability and Testing

• I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)

To do

• 404s on requests to webpack *.chunk.js
  • In GDK, this worked after reconfiguring webpack.incremental to false on both primary and secondary. But in my browser on gitlab.com, various *.chunk.js files are fetched from gitlab.com as well as assets.gitlab-static.net.
  • This doesn't happen when /assets requests are proxied.
• Browser console error Content Security Policy: The page’s settings blocked the loading of a resource at http://gdk.test:3809/sockjs-node/info?t=1622837693220 (“connect-src”).
  • This doesn't happen when /assets requests are proxied. Because then the resource is at http://gdk.test:3808 which is included in the Content-Security-Policy response headers generated by the primary.
  • Idea for later: Somehow add secondary webpack addresses in ::Gitlab::ContentSecurityPolicy::ConfigLoader.allow_webpack_dev_server.
    • I don't see this sockjs-node request from my browser on gitlab.com.
      • Is this used in production?
• 400 Browser console error Firefox can’t establish a connection to the server at ws://gdk.test:3001/-/cable. from commons-pages.groups.epics.index-pages.groups.epics.new-pages.groups.epics.show-pages.groups.iterati-8ee3690e.chunk.js:5839:25 => #333579 (closed)
  • Without this MR on geo.staging.gitlab.com, /-/cable 404s.
  • Without this MR locall, without GEO_SECONDARY_PROXY=1, /-/cable 101s which I guess is good.
• 502 requests. Workhorse error output badgateway: failed to receive response: dial unix /Users/mkozonogitlab/Developer/gdk-geo/gitlab.socket: socket: too many open files. It goes away after restarting gitlab-workhorse then comes back.
  • I thought maybe this was because my WIP code was instantiating a new handler to proxy to the primary for every proxied request. I fixed that but this still happens.
  • I changed the way I close the API call response body and I didn't see this again after browsing around for 5 minutes
• How can I set the environment variable GEO_SECONDARY_PROXY=1 for gdk start gitlab-workhorse? At the moment I run gitlab/workhorse/gitlab-workhorse manually just so I can set this variable.

How to test this change locally on MacOS

• Change directory to your Geo secondary GDK (This feature should have no effect on a Geo primary or non-Geo instance, though even in those cases, Workhorse will begin to make new internal API calls): cd /path/to/your/gdk-geo
• Build workhorse: make gitlab-workhorse-update

After Workhorse is built, then you need to run Workhorse with an environment variable set GEO_SECONDARY_PROXY=1. I don't know why this is so hard, but you have two options:

Option 1: Set GEO_SECONDARY_PROXY=1 in Procfile

• Edit your Geo GDK's Procfile
• Add GEO_SECONDARY_PROXY=1 to the Workhorse line gitlab-workhorse: exec /usr/bin/env so it looks like gitlab-workhorse: exec /usr/bin/env GEO_SECONDARY_PROXY=1 (you only have to do this once until the next reconfigure)
• gdk restart gitlab-workhorse

Option 2: Run gitlab-workhorse bin manually so it can see your shell's environment variables

• Note the command that GDK runs to run workhorse: ps aux | grep gitlab-workhorse
• Stop GDK-managed workhorse since you need to run it with an environment variable, which sounds like a pain to set system-wide on MacOS: gdk stop gitlab-workhorse
• Set the environment variable and run Workhorse manually. This is what I run: GEO_SECONDARY_PROXY=1 ./gitlab/workhorse/gitlab-workhorse -authSocket /Users/mkozonogitlab/Developer/gdk-geo/gitlab.socket -cableSocket /Users/mkozonogitlab/Developer/gdk-geo/gitlab.socket -listenAddr gdk.test:3001 -documentRoot /Users/mkozonogitlab/Developer/gdk-geo/gitlab/public -developmentMode -secretPath /Users/mkozonogitlab/Developer/gdk-geo/gitlab/.gitlab_workhorse_secret -config /Users/mkozonogitlab/Developer/gdk-geo/gitlab/workhorse/config.toml -logFormat json

When running Workhorse with the environment variable set, you should see Workhorse log output like Geo Proxy: Handle this request locally or Geo Proxy: Forward this request.