## Summary OpenBao requests intermittently fail with `api error: status code 504` (~8 occurrences over the last 7 days on production; surfaced to users as runner secret-fetch failures). The root cause is **not** the database — it's that OpenBao's `http` audit device POSTs every request to GitLab Rails **synchronously, on the request critical path, with no timeout and no retry**. When that POST is slow, the request hangs until the GCP load balancer's 30s backend timeout cuts it off and returns a 504. This affects **any OpenBao operation** (read/write/list, auth login, policy/permission ops, provisioning) — the audit step is on the generic request path. Secret fetches are simply the most frequent victim because the scheduled test pipeline exercises them every 5 minutes. The `failed to start underlying postgresql transaction: context canceled` line is a downstream symptom (`BeginTx` failing on an already-canceled context), not a DB problem. ## What happens 1. Caller → CloudFlare → GCP ALB → OpenBao: any API request. 2. Before running the operation, OpenBao audit-logs the request to **both** enabled devices — `file/stdout` (local, instant) and `http/remote` (POST to `gitlab.com/api/v4/internal/secrets_manager/audit_logs`). The audit broker calls every device synchronously and blocks on each. 3. That Rails POST occasionally takes ≥30s. The audit HTTP client uses `http.DefaultClient` (no timeout) and does no retry, so OpenBao just waits. 4. The OpenBao backend service has `timeoutSec = 30`, so the Application Load Balancer (ALB) returns **504** to the caller at 30s and tears down the backend connection. 5. The teardown cancels OpenBao's request context → in-flight ops die with `context canceled` (the audit POST, and `BeginTx` for requests that got past the audit step). ## Evidence - **It's request-path-wide, not read-specific.** The 8 × 504 span KV **reads** (`secret_for_main`, `secret_for_all_branches`), an **auth config** op (`auth/pipeline_jwt/config`), and a **policy list** (`sys/policies/detailed/acl/users?list=true`). A runner secret fetch is itself two exposed requests — the inline JWT **login** and the KV **read**. - **504 = `backend_timeout` at a hard 30s.** LB logs: every 504 has `statusDetails=backend_timeout`, `httpRequest.latency` clamped 30.005–30.105s. Normal requests on the same paths: ~0.4s. Failed job [14701594276](https://gitlab.com/issue-reproduce/gitlab-secrets-manager-group/project-secrets/-/jobs/14701594276) waited 30.2s (18:47:04 → 18:47:34) then 504. - **The 30s source, read live:** `gcloud compute backend-services describe gkegw1-ltbu-secrets-manager--secrets-manager--8200-... --global --project=gitlab-runway-production` → `timeoutSec: 30` (no `timeoutSec` set in config → GKE Gateway default applied). - **DB is idle at every incident** → pool exhaustion / DB load ruled out. Cloud SQL `num_backends` flat at 4, CPU ~4%, normal transaction rate. - **No pod restart / leadership change / seal / OOM** at any incident → HA churn ruled out. - **The pg error lives inside the `jsonPayload.type="response"` audit event**, which is why a plain log-message search doesn't find it. ## "Fail-safe" ≠ non-blocking (important) Audit is fail-safe (fail-closed: don't serve a request unless ≥1 device logs it), and `file/stdout` satisfies that — so `http/remote` is redundant for the guarantee. But the broker still calls every device synchronously and only checks "≥1 succeeded" *after* the loop, so the redundant remote device still blocks the request. Fail-safe protects against a device being *down* (fast error → request proceeds); it does nothing about a device being *slow/hung*. With no client timeout, "slow" is bounded only by the 30s LB cutoff. OpenBao also POSTs and blocks on the audit for every operation even when Rails ultimately discards it (Rails persists only 8 event types and ignores `request`-type events). ## References - https://gitlab.slack.com/archives/C09GSEDT50U/p1780634925096419 - ADR issue - https://gitlab.com/gitlab-org/gitlab/-/work_items/470461 - Broker loops all devices, blocks on each, fail-safe check after the loop: [`vault/audit_broker.go#L103-168`](https://gitlab.com/gitlab-org/govern/secrets-management/openbao-mirror/-/blob/main/vault/audit_broker.go#L103-168) - `http` audit device — synchronous POST on request ctx, `http.DefaultClient`, no retry: [`builtin/audit/http/backend.go#L194-234`](https://gitlab.com/gitlab-org/govern/secrets-management/openbao-mirror/-/blob/main/builtin/audit/http/backend.go#L194-234), [`#L277`](https://gitlab.com/gitlab-org/govern/secrets-management/openbao-mirror/-/blob/main/builtin/audit/http/backend.go#L277), [`#L27`](https://gitlab.com/gitlab-org/govern/secrets-management/openbao-mirror/-/blob/main/builtin/audit/http/backend.go#L27) - Audit runs before the routed operation (generic request path, any op): [`vault/request_handling.go#L1229-1236`](https://gitlab.com/gitlab-org/govern/secrets-management/openbao-mirror/-/blob/main/vault/request_handling.go#L1229-1236) - pg transaction error: [`physical/postgresql/transaction.go#L25-37`](https://gitlab.com/gitlab-org/govern/secrets-management/openbao-mirror/-/blob/main/physical/postgresql/transaction.go#L25-37) ## Docs: - [OpenBao audit](https://openbao.org/docs/audit/) - [OpenBao http device](https://openbao.org/docs/audit/http/) · [Go `database/sql` BeginTx](https://pkg.go.dev/database/sql#DB.BeginTx)