2025-09-24: 500 Errors in gstg Observed in Smoke Tests

500 Errors in gstg Observed in Smoke Tests (Severity 3 (Medium))

Problem: Multiple 500 Internal Server Errors are occurring during API operations in QA smoke tests on both staging and staging canary environments, affecting different repositories and occurring across multiple Gitaly nodes.

Impact: QA smoke tests are failing in both the staging and staging canary environments, which is blocking deployments to production and delaying a planned security release.

Causes: A recent merge request to Gitaly (MR 8013) is suspected to have introduced problems with context propagation during access checks. The failures in QA pipelines match a previously quarantined test. The team is planning to revert this change.

Response strategy: We have escalated the issue to the Gitaly Tier 2 on-call team for urgent investigation, shared relevant job logs, and are planning to revert the recent Gitaly merge request suspected to have caused the failures.

---

This ticket was created to track INC-4141, by incident.io 🔥