Skip to content

Geo push on secondary fails with read-only message in CNG - double-trailing slash for internal API url

When using CNG defaults, the gitlab-shell internal API URL (gitlab_url in the gitlab-shell configmap) has a trailing slash, and pushing through a Geo secondary will make a request with double slashes to the internal API.

This will be rejected by the read-only middleware check, as it verifies whether the path starts with /api/v#{version}/geo/proxy_git_ssh" - and in this case, it'll start with //api/.. and it fails.

Relevant log examples:

time="2022-05-31T03:01:45Z" level=error msg="Internal API error" correlation_id=01G4C0ZV2Y2TDY0Y1ACF3C6PJ0 duration_ms=7ns error="You cannot perform write operations on a read-only instance" method=POST status=403 url="http://gitlab-webservice-default.default.svc:8181//api/v4/geo/proxy_git_ssh/info_refs_receive_pack`
{"severity":"DEBUG","time":"2022-05-31T03:01:45.775Z","correlation_id":"01G4C0ZV5996AEWTZEN0QSBTJN","message":"GitLab ReadOnly: preventing possible non read-only operation"}

We seemed to use to strip trailing slashes from the internal API URL in gitlab-shell (gitlab-shell!35 (merged)) but that changed at some point as it seems it's passed as-is from the config now.

Opening this issue in the main gitlab-org/gitlab repo as it's not very clear where the problem is yet (charts, gitlab-shell, the read-only middleware check in Rails)

There are multiple possible fixes:

  1. Current "simplest" workaround seems to be removing the trailing slash in the gitlab-shell configmap in CNG; and/or potentially update the default CNG configmap to remove the trailing /
  2. Remove trailing slashes in gitlab-shell directly, so the URL can be passed in with or without trailing slashes - requires some double checking to make sure nothing else breaks, or is expecting a trailing slash here
  3. Update the Rails geo_proxy_git_ssh_route? method to handle multiple leading slashes