Internal server error when loading project with many pipelines
Summary
Loading Project Details fails with 500 Internal Error when the last commit has too many pipelines.
Steps to reproduce
-
Make the latest commit on the main branch have enough pipelines to make the SQL query timeout. (On our setup, that amount is ~20400 pipelines).
-
Try to access the Project Details page.
-
Timeout + 500 error.
An easy way to reproduce is to reduce the db timeout. It will require less pipelines to reproduce.
What is the current bug behavior?
The main project view is inaccesible due to many pipelines on the latest main branch commit.
What is the expected correct behavior?
The project should be always accessible and not be slowed down due to many pipelines.
Other views like the detailed list of pipelines on the commit view is correctly accessible.
Relevant logs and/or screenshots
Running on self hosted GitLab Community Edition 12.1.1.
Started GET "/cki-project/pipeline-trigger-data.git/info/refs?service=git-upload-pack" for [ip] at 2019-11-11 14:40:25 +0000
Processing by Projects::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"cki-project", "project_id"=>"pipeline-trigger-data.git"}
Completed 500 Internal Server Error in 60386ms (ActiveRecord: 60017.5ms)
ActionView::Template::Error (PG::QueryCanceled: ERROR: canceling statement due to statement timeout
: SELECT "ci_pipelines"."sha", "ci_pipelines"."status" FROM "ci_pipelines" LEFT OUTER JOIN "ci_pipelines" "ci_pipelines_2" ON "ci_pipelines"."sha" = "ci_pipelines_2"."sha" AND "ci_pipelines"."project_id" = "ci_pipelines_2"."project_id" AND "ci_pipelines"."id" < "ci_pipe$
ines_2"."id" AND "ci_pipelines"."ref" = "ci_pipelines_2"."ref" WHERE "ci_pipelines"."project_id" = 26 AND ("ci_pipelines"."config_source" IN (1, 2) OR "ci_pipelines"."config_source" IS NULL) AND "ci_pipelines"."sha" = '3237b06d153aa2b3ee07576c6fddb30d924f7063' AND "ci_p$
pelines_2"."id" IS NULL AND "ci_pipelines"."ref" = 'master'):
6: - merge_request = local_assigns.fetch(:merge_request, nil)
7: - project = local_assigns.fetch(:project) { merge_request&.project }
8: - ref = local_assigns.fetch(:ref) { merge_request&.source_branch }
9: - commit_status = commit.present(current_user: current_user).status_for(ref)
10:
11: - link = commit_path(project, commit, merge_request: merge_request)
12:
config/initializers/peek.rb:18:in `async_exec_params'
config/initializers/peek.rb:18:in `exec_params'
app/models/ci/pipeline.rb:293:in `each_with_object'
app/models/ci/pipeline.rb:293:in `latest_status_per_commit'
lib/gitlab/metrics/instrumentation.rb:161:in `block in latest_status_per_commit'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `latest_status_per_commit'
app/models/commit.rb:317:in `status_for_project'
app/models/commit.rb:313:in `status'
app/presenters/commit_presenter.rb:9:in `status_for'
app/views/projects/commits/_commit.html.haml:9:in `_app_views_projects_commits__commit_html_haml___4425144487616811625_69865739616100'
app/views/shared/_commit_well.html.haml:4:in `_app_views_shared__commit_well_html_haml___689382185668596160_69865739471000'
app/views/projects/_files.html.haml:15:in `_app_views_projects__files_html_haml__2956979301631742574_69865733977480'
app/views/projects/show.html.haml:34:in `_app_views_projects_show_html_haml__1159162008836357442_69865723885420'
app/controllers/application_controller.rb:117:in `render'
app/controllers/application_controller.rb:522:in `block in allow_gitaly_ref_name_caching'
lib/gitlab/gitaly_client.rb:300:in `allow_ref_name_caching'
app/controllers/application_controller.rb:521:in `allow_gitaly_ref_name_caching'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:445:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:439:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Completed 200 OK in 32ms (Views: 0.3ms | ActiveRecord: 4.2ms)
Possible fixes
The pipelines query should be limited or only that single information shouldn't be shown if the query failed, as the pipelines are not even shown on that view.