Cross-Database Modification Error Causing Pipeline Timeouts

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

The Developer Experience (DevEx) team is working on improving pipeline stability EPIC and has identified a recurring cross-database modification error that's causing significant job timeouts. We need assistance from the Database team to resolve this issue.

Problem Description

We're encountering Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification::CrossDatabaseModificationAcrossUnsupportedTablesError That causes jobs to run for over 60 minutes before timing out.

Error Details

Here is the error message

Rack app ("POST /api/graphql" - (127.0.0.1)): #<Gitlab::Database::QueryAnalyzers::PreventCrossDatabaseModification::CrossDatabaseModificationAcrossUnsupportedTablesError:"Cross-database data modification of 'gitlab_ci_cell_local, gitlab_main_cell' were detected within a transaction modifying the 'ci_runners' tables. \n\nPlease refer to https://docs.gitlab.com/ee/development/database/multiple_databases.html#removing-cross-database-transactions for details on how to resolve this exception.\n\n0: UPDATE \"ci_runners\" SET \"active\" = $1 WHERE \"ci_runners\".\"id\" = $2 /*application:test,correlation_id:01JZV644AAJAQMETVKAEB9EV88,endpoint_id:GraphqlController#execute,db_config_database:gitlabhq_test_ci,db_config_name:ci,line:/app/services/ci/runners/update_runner_service.rb:18:in `execute'*/">

Impact

• Frequency: 47 occurrences in the last 28 days
• Duration: Jobs hang for 60+ minutes before timing out
• Pipeline Impact: ~17% of all Uncategorized job timeout failures are due to this issue

Related Issue

gitlab-org/quality/analytics/team#145

Related Links

Here the list of failed jobs , due to same error

https://gitlab.com/gitlab-org/gitlab/-/jobs/11034201497

-------- 11 August 2025 JOB URL's ----------
https://gitlab.com/gitlab-org/gitlab/-/jobs/10960254799
https://gitlab.com/gitlab-org/gitlab/-/jobs/10960994892
https://gitlab.com/gitlab-org/gitlab/-/jobs/10960584697
https://gitlab.com/gitlab-org/gitlab/-/jobs/10959904767
https://gitlab.com/gitlab-org/gitlab/-/jobs/10961572964
https://gitlab.com/gitlab-org/gitlab/-/jobs/10959603999
https://gitlab.com/gitlab-org/gitlab/-/jobs/10960495741
https://gitlab.com/gitlab-org/gitlab-foss/-/jobs/10927991114
https://gitlab.com/gitlab-org/gitlab-foss/-/jobs/10928009387
https://gitlab.com/gitlab-org/gitlab/-/jobs/10912626394
https://gitlab.com/gitlab-org/gitlab/-/jobs/10912780150
https://gitlab.com/gitlab-org/gitlab/-/jobs/10911690477
https://gitlab.com/gitlab-org/gitlab/-/jobs/10912524021
https://gitlab.com/gitlab-org/gitlab/-/jobs/10913144040
https://gitlab.com/gitlab-org/gitlab/-/jobs/10911549692
https://gitlab.com/gitlab-org/gitlab/-/jobs/10911629719
https://gitlab.com/gitlab-org/gitlab/-/jobs/10911266216
https://gitlab.com/gitlab-org/gitlab/-/jobs/10910846908
https://gitlab.com/gitlab-org/gitlab/-/jobs/10912588574
https://gitlab.com/gitlab-org/gitlab/-/jobs/10911201900


--------- 1 AUGUST 2025 JOB URL's ---------------------
https://gitlab.com/gitlab-org/gitlab/-/jobs/10643355456
https://gitlab.com/gitlab-org/gitlab/-/jobs/10643189156 
https://gitlab.com/gitlab-org/gitlab/-/jobs/10642776111
https://gitlab.com/gitlab-org/gitlab/-/jobs/10629853838
https://gitlab.com/gitlab-org/gitlab/-/jobs/10629410539
https://gitlab.com/gitlab-org/gitlab/-/jobs/10629754504
https://gitlab.com/gitlab-org/gitlab/-/jobs/10629397458
https://gitlab.com/gitlab-org/gitlab/-/jobs/10628388634
https://gitlab.com/gitlab-org/gitlab/-/jobs/10629080378
https://gitlab.com/gitlab-org/gitlab/-/jobs/10628985532

-------------------- OLDER JOBS URL -----------------------
https://gitlab.com/gitlab-org/gitlab/-/jobs/10628250701
https://gitlab.com/gitlab-org/gitlab/-/jobs/10617038401
https://gitlab.com/gitlab-org/gitlab/-/jobs/10628196449
https://gitlab.com/gitlab-org/gitlab/-/jobs/10616964655
https://gitlab.com/gitlab-org/gitlab/-/jobs/10560722357
https://gitlab.com/gitlab-org/gitlab/-/jobs/10559993408
https://gitlab.com/gitlab-org/gitlab/-/jobs/10560116230
https://gitlab.com/gitlab-org/gitlab/-/jobs/10509152192              
https://gitlab.com/gitlab-org/gitlab/-/jobs/10509048566
https://gitlab.com/gitlab-org/gitlab/-/jobs/10508859163