Skip to content

GPRD workers often fail to get batches of things to process due to PG::ConnectionFailure: ERROR: server closed the connection unexpectedly

https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+%22PG%3A%3AConnectionFailure%22

E.g. this is a failure to get batch of unsynced projects: https://sentry.gitlap.com/gitlab/gitlabcom/issues/183070/. I also see errors in similar queries getting batches of uploads, lfs objects, and job artifacts.

PG::ConnectionFailure: ERROR: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. CONTEXT: Remote SQL command: FETCH 1000 FROM c1 : SELECT "gitlab_secondary"."projects"."id" FROM "gitlab_secondary"."projects" LEFT OUTER JOIN project_registry ON project_registry.project_id = gitlab_secondary.projects.id WHERE "project_registry"."project_id" IS NULL AND "gitlab_secondary"."projects"."repository_storage" = 'nfs-file05' AND ("gitlab_secondary"."projects"."id" NOT IN (1694245, 3015259, 1727448, 3214806, 2836721, 2839998, 3220837, 1551167)) ORDER BY "gitlab_secondary"."projects"."last_repository_updated_at" DESC LIMIT 1000

Reproducible in console:

mkozono@sidekiq-besteffort-03-sv-gprd.c.gitlab-production.internal:~$ sudo gitlab-rails console
Loading production environment (Rails 4.2.10)
irb(main):001:0> unsynced_projects = Geo::ProjectRegistryFinder.new(current_node: Gitlab::Geo.current_node).find_unsynced_projects(batch_size: 10).to_a;1
WARNING:  no connection to the server

ActiveRecord::StatementInvalid: PG::ConnectionFailure: ERROR:  server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

CONTEXT:  Remote SQL command: FETCH 1000 FROM c1
: SELECT  "gitlab_secondary"."projects".* FROM "gitlab_secondary"."projects" LEFT OUTER JOIN project_registry ON project_registry.project_id = gitlab_secondary.projects.id WHERE "project_registry"."project_id" IS NULL LIMIT 10
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-pg-1.3.0/lib/peek/views/pg.rb:17:in `async_exec'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/peek-pg-1.3.0/lib/peek/views/pg.rb:17:in `async_exec'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/querying.rb:39:in `find_by_sql'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:639:in `exec_queries'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:515:in `load'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.10/lib/active_record/relation.rb:243:in `to_a'
	from (irb):1
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/commands/console.rb:110:in `start'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/commands/console.rb:9:in `start'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:68:in `console'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
	from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/railties-4.2.10/lib/rails/commands.rb:17:in `<top (required)>'
	from bin/rails:14:in `require'
	from bin/rails:14:in `<main>'