Skip to content

Fix timeout issue for rake task gitlab:backup:create

Jeroen Nijhof requested to merge backup-database-timeout-fix into master

This is a fix for a database timeout which can ocure when the backup create task is taking very long (1-2 hours). It seems that ActiveRecord is loosing the connection after a hour idleness and need to be reconnected before use.

I, [2016-02-08T00:45:08.640778 #35165]  INFO -- : ** [Raven] Raven 0.15.4 ready to catch errors
rake aborted!
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() could not receive data from server: Connection timed out
:               SELECT COUNT(*)
              FROM pg_class c
              LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
              WHERE c.relkind IN ('r','v','m') -- (r)elation/table, (v)iew, (m)aterialized view
              AND c.relname = 'schema_migrations'
              AND n.nspname = ANY (current_schemas(false))
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:105:in `table_exists?'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/migration.rb:846:in `get_all_versions'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/migration.rb:854:in `current_version'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:6:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'
PG::ConnectionBad: PQconsumeInput() could not receive data from server: Connection timed out
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/postgresql/schema_statements.rb:105:in `table_exists?'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/migration.rb:846:in `get_all_versions'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.1/lib/active_record/migration.rb:854:in `current_version'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:6:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)

/cc @jacobvosmaer @DouweM

Merge request reports