Gitaly pre-receive hook sends absolute disk path as project in /internal/allowed causing SSH push failure on Secondary

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Close this issue

Summary

During an SSH push to a Geo Secondary node, the Gitaly pre-receive hook sends an absolute filesystem path as the project parameter in the /api/v4/internal/allowed request to Primary. Within the same push flow, the earlier GitLab Shell request correctly sends the canonical project path. This inconsistency, combined with quarantine context leaking cross-node, causes the push to fail with 500 Internal Server Error.

Two distinct defects identified in this flow:

Gitaly bug: AllowedParams struct lacks a GlProjectPath field, causing the HTTP client to use the absolute disk path as the project parameter
Geo architectural issue: The pre-receive hook on Secondary sends quarantine environment variables to Primary Rails, which then passes them to Primary's Gitaly. Primary's Gitaly cannot access Secondary's quarantine objects, causing cat-file failure. This is already logged in #598328

Steps to reproduce

Set up GitLab Geo with Primary and at least one Secondary node (unified URL setup)
Ensure both nodes have the same external_url and identical Gitaly storage paths
Configure Secondary's gitlab_url (gitlab-shell) and [gitlab] url (gitaly) to point to Primary
SSH directly to a Secondary node and push:
```
git push origin feat/new-branch
```
Push fails with pre-receive hook declined / 500 Internal Server Error
Control: push directly to Primary via SSH succeeds

What is the current bug behavior?

Within a single SSH push, two /internal/allowed requests are sent with different project formats:

Request 1 (GitLab Shell, pre-check) — uses canonical project path:

{ "project": "namespace/project.git", "changes": "_any" }

Request 2 (Gitaly pre-receive hook, actual push) — uses absolute disk path:

{ "project": "/data/repositories/@hashed/ab/cd/<hash>.git" }

Primary Rails forwards quarantine context from Secondary to its local Gitaly, which fails because the quarantine objects only exist on Secondary's filesystem:

fatal: not a git repository: '/data/repositories/@hashed/ab/cd/<hash>.git'

What is the expected correct behavior?

The Gitaly pre-receive hook should send the canonical project path (e.g., namespace/project.git) or use gl_project_path (which is already available on the Repository proto) — not the absolute filesystem path
Quarantine context from one Geo node should not leak to another node's Gitaly access checks
SSH push through Geo Secondary should succeed, either by proxying to Primary or by handling the cross-node context correctly

Relevant logs and/or screenshots

Gitaly log on Secondary — note glProjectPath is correct but not used for project:

{
  "error": "500 Internal Server Error",
  "method": "POST",
  "url": "http://<primary_ip>/api/v4/internal/allowed",
  "grpc.method": "PreReceiveHook",
  "grpc.request.glProjectPath": "namespace/project",
  "grpc.request.glRepository": "project-NNNN",
  "grpc.request.repoPath": "@hashed/ab/cd/<hash>.git"
}

API log on Primary — Request 1 (200 OK):

{
  "status": 200,
  "params": {
    "action": "git-receive-pack",
    "project": "namespace/project.git",
    "changes": "_any",
    "protocol": "ssh"
  }
}

API log on Primary — Request 2 (500 Error):

{
  "status": 500,
  "params": {
    "action": "git-receive-pack",
    "project": "/data/repositories/@hashed/ab/cd/<hash>.git",
    "relative_path": "@hashed/ab/cd/<hash>.git",
    "protocol": "ssh"
  },
  "exception.class": "GRPC::Internal",
  "exception.message": "13:cat-file failed: exit status 128, stderr: \"fatal: not a git repository: '/data/repositories/@hashed/ab/cd/<hash>.git'\""
}

Root cause analysis

Defect 1: Wrong `project` parameter in Gitaly pre-receive hook

gitaly/internal/gitlab/client.go:8-29 — AllowedParams struct has no GlProjectPath field:

type AllowedParams struct {
    RepoPath     string // absolute path — used as Project
    RelativePath string
    GLRepository string
    // GlProjectPath is MISSING
    ...
}

gitaly/internal/gitaly/hook/prereceive.go:103,135-136 — sets RepoPath to absolute disk path:

repoPath, err := m.locator.GetRepoPath(ctx, repo)  // returns /data/repositories/@hashed/...
// ...


params := gitlab.AllowedParams{
    RepoPath: repoPath,  // absolute filesystem path
    // repo.GetGlProjectPath() is available but not used
}

gitaly/internal/gitlab/http_client.go:183 — sends absolute path as project:

req := allowedRequest{
    Project: strings.Replace(params.RepoPath, "'", "", -1),  // absolute path!
}

Note: gl_repository IS sent alongside project, so Rails can still find the project via parse_repo_path. But the wrong project violates the API contract.

Defect 2: Quarantine context leaking across Geo nodes

The pre-receive hook on Secondary sends quarantine env vars (GIT_OBJECT_DIRECTORY_RELATIVE, GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE) to Primary's /internal/allowed.

Primary Rails stores these in HookEnv (lib/api/internal/base.rb:87):

Gitlab::Git::HookEnv.set(gl_repository, params[:relative_path], parse_env) if container

Then during access check, GitalyClient::Util.repository() (lib/gitlab/gitaly_client/util.rb:7-19) embeds these quarantine paths into every Gitaly RPC:

git_env = Gitlab::Git::HookEnv.all(gl_repository)
git_object_directory = git_env['GIT_OBJECT_DIRECTORY_RELATIVE'].presence
# ...embedded into Gitaly::Repository proto

Primary's Gitaly receives quarantine paths that only exist on Secondary → cat-file fails.

Why `geo_proxy_push_ssh_to_primary` doesn't help

ee/lib/ee/gitlab/geo_git_access.rb:30-32:

def forward_ssh_git_request_to_primary?
  return false unless protocol == 'ssh'
  return false unless ::Gitlab::Database.read_only?  # ← fails on Primary

When Secondary's GitLab Shell sends /internal/allowed directly to Primary Rails, Gitlab::Database.read_only? returns false (Primary is read-write). The SSH proxy flow is never triggered. This feature assumes GitLab Shell talks to local (Secondary) Rails.

Full failure chain

1. SSH → Secondary sshd → Secondary GitLab Shell
2. Shell → Primary Rails /internal/allowed (project: "ns/proj.git") → 200 OK
   └─ Response: Gitaly address = unix:///var/opt/gitlab/gitaly/gitaly.socket
3. Shell → Secondary Gitaly (Unix socket resolves to local Gitaly!)
4. Secondary Gitaly: git receive-pack → pre-receive hook fires
5. Hook → Primary Rails /internal/allowed
   ├─ project: "/data/repositories/@hashed/..." (wrong)
   ├─ gl_repository: "project-NNNN" (correct, Rails uses this)
   └─ env: quarantine dirs from Secondary (wrong node)
6. Primary Rails → HookEnv stores Secondary's quarantine paths
7. Primary Rails → access check → calls Primary Gitaly with Secondary's quarantine
8. Primary Gitaly: quarantine dirs don't exist → cat-file fails → GRPC::Internal → 500

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

GitLab Version: 18.8.x-ee (Omnibus) GitLab Shell: 14.45.5 Ruby: 3.2.8 PostgreSQL: 16.11 OS: Ubuntu 22.04 Geo: Enabled (Primary + 2 Secondary) Feature flag geo_secondary_proxy: true

Possible fixes

Fix 1 (Gitaly — project parameter):

Add GlProjectPath string field to AllowedParams in gitaly/internal/gitlab/client.go
Pass repo.GetGlProjectPath() in gitaly/internal/gitaly/hook/prereceive.go
Use GlProjectPath for Project field in gitaly/internal/gitlab/http_client.go (fall back to RepoPath if empty)

Fix 2 (Rails — quarantine context isolation):

Detect cross-node /internal/allowed requests (e.g., from Geo Secondary) and skip setting quarantine HookEnv, OR
Clear quarantine context before running access checks for hook-originated requests from remote nodes

Fix 3 (Geo SSH proxy) is in a separate issue for #598328.

Impact

SSH push through Geo Secondary is completely broken when the push lands on a Secondary node
Affects any Geo deployment using unified URL with SSH access behind a load balancer
Workaround: route SSH traffic (port 22) exclusively to Primary node at the LB level

work_item#466057 — push SSH proxy to primary
Feature flags: geo_proxy_fetch_ssh_to_primary, geo_proxy_push_ssh_to_primary
Affected components: Gitaly pre-receive hook, GitLab Shell, Internal API, Geo
Affected endpoint: POST /api/v4/internal/allowed

Edited Apr 28, 2026 by Natanael Silva