Fixed. `ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server` when backup heavy repository (!16598) · Merge requests · GitLab.org / GitLab

The source project of this merge request has been removed.

sue445 requested to merge (removed):backup_repository-with_reconnect_and_retry into master Sep 13, 2019

This MR is moved from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32638

Followings are original description

What does this MR do?

rake gitlab:backup:repo:create is failed immediately after backup the heavy repository. (about 23GB)

Backtrace

* my-group/my-light-repo  ... [DONE]
[SKIPPED] Wiki
* my-group/my-heavy-repo ...
[WARNING] Executing PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
       This probably means the server terminated abnormally
       before or while processing the request.
: SELECT "features"."key" FROM "features"
Ignoring error on my-group/my-heavy-repo - Failed to backup repo
[DONE]
rake aborted!
ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server
: SELECT "features"."key" FROM "features"
/home/git/gitlab/config/initializers/peek.rb:18:in `async_exec_params'
/home/git/gitlab/config/initializers/peek.rb:18:in `exec_params'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:624:in `block (2 levels) in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:46:in `block in permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:45:in `permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:623:in `block in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:46:in `block in permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:45:in `permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:623:in `block in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:613:in `block (2 levels) in log'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:612:in `block in log'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:604:in `log'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:622:in `exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:609:in `execute_and_clear'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_query'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:375:in `select_prepared'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:40:in `select_all'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/query_cache.rb:97:in `select_all'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/calculations.rb:176:in `pluck'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/querying.rb:14:in `pluck'
/home/git/gitlab/lib/feature.rb:13:in `feature_names'
/home/git/gitlab/lib/feature.rb:33:in `persisted_names'
/home/git/gitlab/lib/feature.rb:40:in `persisted?'
/home/git/gitlab/lib/feature.rb:52:in `enabled?'
/home/git/gitlab/lib/feature/gitaly.rb:19:in `enabled?'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:222:in `request_kwargs'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:155:in `call'
/home/git/gitlab/lib/gitlab/gitaly_client/repository_service.rb:19:in `exists?'
/home/git/gitlab/lib/gitlab/git/repository.rb:90:in `exists?'
/home/git/gitlab/app/models/repository.rb:567:in `exists?'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:41:in `block (2 levels) in cache_method_asymmetrically'
/home/git/gitlab/lib/gitlab/repository_cache.rb:43:in `fetch_without_caching_false'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:114:in `block (2 levels) in cache_method_output_asymmetrically'
/home/git/gitlab/lib/gitlab/null_request_store.rb:34:in `fetch'
/home/git/gitlab/lib/gitlab/safe_request_store.rb:12:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache.rb:24:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:113:in `block in cache_method_output_asymmetrically'
/home/git/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:127:in `block in memoize_method_output'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:136:in `no_repository_fallback'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:126:in `memoize_method_output'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:112:in `cache_method_output_asymmetrically'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:40:in `block in cache_method_asymmetrically'
/home/git/gitlab/app/models/repository.rb:575:in `empty?'
/home/git/gitlab/app/models/repository.rb:406:in `expire_emptiness_caches'
/home/git/gitlab/lib/backup/repository.rb:157:in `empty_repo?'
/home/git/gitlab/lib/backup/repository.rb:35:in `block in dump'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:230:in `block in in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `loop'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:62:in `find_each'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/querying.rb:9:in `find_each'
/home/git/gitlab/lib/backup/repository.rb:17:in `dump'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:88:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
                                                                                                                                                                                                                    [13/1238]
Caused by:
PG::UnableToSend: no connection to the server
/home/git/gitlab/config/initializers/peek.rb:18:in `async_exec_params'
/home/git/gitlab/config/initializers/peek.rb:18:in `exec_params'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:624:in `block (2 levels) in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:46:in `block in permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/concurrency/share_lock.rb:185:in `yield_shares'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/dependencies/interlock.rb:45:in `permit_concurrent_loads'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:623:in `block in exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:613:in `block (2 levels) in log'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:612:in `block in log'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.1.7/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract_adapter.rb:604:in `log'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:622:in `exec_no_cache'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:609:in `execute_and_clear'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_query'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:375:in `select_prepared'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/database_statements.rb:40:in `select_all'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/connection_adapters/abstract/query_cache.rb:97:in `select_all'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/calculations.rb:176:in `pluck'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/querying.rb:14:in `pluck'
/home/git/gitlab/lib/feature.rb:13:in `feature_names'
/home/git/gitlab/lib/feature.rb:33:in `persisted_names'
/home/git/gitlab/lib/feature.rb:40:in `persisted?'
/home/git/gitlab/lib/feature.rb:52:in `enabled?'
/home/git/gitlab/lib/feature/gitaly.rb:19:in `enabled?'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:222:in `request_kwargs'
/home/git/gitlab/lib/gitlab/gitaly_client.rb:155:in `call'
/home/git/gitlab/lib/gitlab/gitaly_client/repository_service.rb:19:in `exists?'
/home/git/gitlab/lib/gitlab/git/repository.rb:90:in `exists?'
/home/git/gitlab/app/models/repository.rb:567:in `exists?'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:41:in `block (2 levels) in cache_method_asymmetrically'
/home/git/gitlab/lib/gitlab/repository_cache.rb:43:in `fetch_without_caching_false'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:114:in `block (2 levels) in cache_method_output_asymmetrically'
/home/git/gitlab/lib/gitlab/null_request_store.rb:34:in `fetch'
/home/git/gitlab/lib/gitlab/safe_request_store.rb:12:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache.rb:24:in `fetch'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:113:in `block in cache_method_output_asymmetrically'
/home/git/gitlab/lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:127:in `block in memoize_method_output'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:136:in `no_repository_fallback'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:126:in `memoize_method_output'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:112:in `cache_method_output_asymmetrically'
/home/git/gitlab/lib/gitlab/repository_cache_adapter.rb:40:in `block in cache_method_asymmetrically'
/home/git/gitlab/app/models/repository.rb:575:in `empty?'
/home/git/gitlab/app/models/repository.rb:406:in `expire_emptiness_caches'
/home/git/gitlab/lib/backup/repository.rb:157:in `empty_repo?'
/home/git/gitlab/lib/backup/repository.rb:35:in `block in dump'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `each'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:63:in `block in find_each'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:230:in `block in in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `loop'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:214:in `in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.1.7/lib/active_record/querying.rb:9:in `find_each'
/home/git/gitlab/lib/backup/repository.rb:17:in `dump'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:88:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:11:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.6.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:repo:create
(See full trace by running task with --trace)

Error line are followings.

my-group/my-heavy-repo.bundle was created in about 30 minutes, but connection was lost (maybe).

So empty_repo?(wiki) was failed.

Versions

gitlab-ce 12.0.3
pgbouncer 1.10.0
PostgreSQL 10.9

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Changelog entry for user-facing changes, or community contribution. Check the link for other scenarios.
Documentation created/updated or follow-up review issue created
Code review guidelines
Merge request performance guidelines
Style guides
Database guides
Separation of EE specific content

Performance and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
Tested in all supported browsers

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Admin message

Fixed. `ActiveRecord::StatementInvalid: PG::UnableToSend: no connection to the server` when backup heavy repository