Geo: Secondary site returns 500 errors due to CI minutes tracking write attempts on read-only database
Summary
Geo secondary sites return 500 errors when browsing certain projects because Ci::Minutes::NamespaceMonthlyUsage.find_or_create_current attempts to INSERT into the read-only database.
Root Cause
The CI minutes tracking code path is not Geo-aware. When rendering the shared runners minutes limit alert, it attempts to write to ci_namespace_monthly_usages which fails on secondary sites.
Steps to reproduce
- Set up Geo primary and secondary
- Browse a project on secondary where CI minutes alert renders and the usage record doesn't exist
- 500 error occurs
Workaround: Browse the project on primary first to create the record.
Technical Details
Exception: PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only transaction
Failed SQL:
INSERT INTO "ci_namespace_monthly_usages" ("namespace_id", "date", "created_at")
VALUES ($1, $2, $3) RETURNING "id"
Stack Trace
ee/app/models/ci/minutes/namespace_monthly_usage.rb:47 in `block in find_or_create_current`
ee/app/models/ci/minutes/namespace_monthly_usage.rb:36 in `find_or_create_current`
ee/app/models/ci/minutes/usage.rb:71 in `current_usage`
ee/app/models/ci/minutes/usage.rb:38 in `total_minutes_used`
ee/app/models/ci/minutes/usage.rb:34 in `current_balance`
ee/app/models/ci/minutes/usage.rb:79 in `total_minutes_remaining`
ee/app/models/ci/minutes/usage.rb:30 in `percent_total_minutes_remaining`
ee/app/models/ci/minutes/context.rb:8 in `percent_total_minutes_remaining`
ee/app/models/ci/minutes/notification.rb:44 in `percentage`
ee/app/models/ci/minutes/notification.rb:75 in `calculate_notification_stage`
ee/app/models/ci/minutes/notification.rb:15 in `initialize`
ee/app/components/namespaces/pipelines/alerts/shared_runners_minutes_limit_alert_component.rb:18 in `initialize`
ee/app/views/shared/_shared_runners_minutes_limit.html.haml:5
app/views/projects/_flash_messages.html.haml:14
app/views/projects/empty.html.haml:6
app/controllers/projects_controller.rb:429 in `render_landing_page`
app/controllers/projects_controller.rb:181 in `show`
Environment
- GitLab version: 18.5.5
- Deployment: Omnibus multi-node with Praefect
- Site: Secondary
Implementation Plan
Skip the code that would perform a DB write, if Gitlab::Geo.secondary? . There are other examples in the codebase.
Edited by 🤖 GitLab Bot 🤖