2026-08-25: 500s in ops.gitlab.net

500s in ops.gitlab.net (Severity 2)

Problem: The ops production database was written to by a tool that should have been targeting a non-production database, causing ops.gitlab.net to return 500s and timeouts.

Impact: Ops.gitlab.net has been returning 500s and timeouts, which is blocking internal operations including deploys, chatops feature flag changes, auto-deploy, and releases.

Causes: A tool intended for a non-production database wrote to the ops production database instead, leaving it in an unclear state.

Response strategy: We restored the database from the morning backup into a new Cloud SQL instance in us-east1, switched ops.gitlab.net to that restored instance, applied the required cloud-sql-proxy configuration change, scaled the affected workloads back up, and re-enabled ArgoCD auto-sync.