PostgreSQL lag_too_great check fails with inactive and unused replication slots

Summary

BackgroundMigrationWorker tasks failed

Steps to reproduce

• PostgreSQL database
• usage of replication slots
• create NEW replication slot and doesn't use it. It won't have restart_lsn and active = false.

Wait for BackgroundMigrationWorker task to fail

What is the current bug behavior?

BackgroundMigrationWorker tasks fails to run

What is the expected correct behavior?

BackgroundMigrationWorker task should not fail

Relevant logs and/or screenshots

Error log

2019-06-24T06:25:25.666Z 19481 TID-ovn71tgh5 BackgroundMigrationWorker JID-2dcaa75243751f0935737921 INFO: start 2019-06-24T06:25:25.681Z 19481 TID-ovn71tgh5 BackgroundMigrationWorker JID-2dcaa75243751f0935737921 INFO: fail: 0.015 sec 2019-06-24T06:25:25.681Z 19481 TID-ovn71tgh5 WARN: {"context":"Job raised exception","job":{"class":"BackgroundMigrationWorker","args":["Migra teLegacyArtifacts",[12,81]],"retry":3,"queue":"background_migration","jid":"2dcaa75243751f0935737921","created_at":1561357445.5932164,"correla tion_id":"3d32e46d-44df-4eb4-9f0d-f114ad209ed4","enqueued_at":1561357525.6633387},"jobstr":"{\"class\":\"BackgroundMigrationWorker\",\"args\": [\"MigrateLegacyArtifacts\",[12,81]],\"retry\":3,\"queue\":\"background_migration\",\"jid\":\"2dcaa75243751f0935737921\",\"created_at\":156135 7445.5932164,\"correlation_id\":\"3d32e46d-44df-4eb4-9f0d-f114ad209ed4\",\"enqueued_at\":1561357525.6633387}"} 2019-06-24T06:25:25.682Z 19481 TID-ovn71tgh5 WARN: NoMethodError: undefined method `>=' for nil:NilClass 2019-06-24T06:25:25.683Z 19481 TID-ovn71tgh5 WARN: /data/git/gitlab/app/models/postgresql/replication_slot.rb:31:in `block in lag_too_great?' /data/git/gitlab/app/models/postgresql/replication_slot.rb:31:in `count' /data/git/gitlab/app/models/postgresql/replication_slot.rb:31:in `lag_too_great?' /data/git/gitlab/app/workers/background_migration_worker.rb:81:in `healthy_database?' /data/git/gitlab/app/workers/background_migration_worker.rb:50:in `perform_and_ttl' /data/git/gitlab/app/workers/background_migration_worker.rb:23:in `perform' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:185:in `execute_job' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:167:in `block (2 levels) in process' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:128:in `block in invoke' /data/git/gitlab/lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' /data/git/gitlab/lib/gitlab/sidekiq_middleware/correlation_logger.rb:10:in `block in call' /data/git/gitlab/lib/gitlab/correlation_id.rb:15:in `use_id' /data/git/gitlab/lib/gitlab/sidekiq_middleware/correlation_logger.rb:9:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' /data/git/gitlab/lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' /data/git/gitlab/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' /data/git/gitlab/lib/gitlab/sidekiq_middleware/shutdown.rb:54:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sentry-raven-2.7.4/lib/raven/integrations/sidekiq.rb:9:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/middleware/chain.rb:133:in `invoke' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:166:in `block in process' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:108:in `local' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:42:in `block in call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/execution_wrapper.rb:85:in `wrap' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/reloader.rb:68:in `block in wrap' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/execution_wrapper.rb:85:in `wrap' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-5.0.7.1/lib/active_support/reloader.rb:67:in `wrap' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/rails.rb:41:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:243:in `stats' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/job_logger.rb:8:in `call' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/job_retry.rb:73:in `global' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:125:in `block in dispatch' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/logging.rb:48:in `with_context' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/logging.rb:42:in `with_job_hash_context' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:124:in `dispatch' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:165:in `process' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:83:in `process_one' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/processor.rb:71:in `run' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/util.rb:16:in `watchdog' /data/git/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.3/lib/sidekiq/util.rb:25:in `block in safe_thread'

Output of checks

Results of GitLab environment info

System information System: Ubuntu 16.04 Current User: git Using RVM: no Ruby Version: 2.5.5p157 Gem Version: 2.7.6.2 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.0.6 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: go1.10.3 linux/amd64

GitLab information Version: 11.11.3 Revision: e3eeb779 Directory: /data/git/gitlab DB Adapter: PostgreSQL DB Version: 9.6.13 URL: https://gitlab.serv-ac.com HTTP Clone URL: https://gitlab.serv-ac.com/some-group/some-project.git SSH Clone URL: git@git.serv-ac.com:some-group/some-project.git Using LDAP: no Using Omniauth: no

GitLab Shell Version: 9.1.0 Repository storage paths:

• default: /data/git/repositories GitLab Shell path: /data/git/gitlab-shell Git: /usr/bin/git

Results of GitLab application Check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0) Running /data/git/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK

Access to /home/git/.ssh/authorized_keys: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors Projects have namespace: ... 18/11 ... yes 18/12 ... yes 2/13 ... yes 3/14 ... yes 3/15 ... yes 3/16 ... yes 3/17 ... yes 3/18 ... yes 3/19 ... yes 3/20 ... yes 3/21 ... yes 3/22 ... yes 3/23 ... yes 3/24 ... yes 18/25 ... yes 20/27 ... yes 20/29 ... yes 18/30 ... yes 24/31 ... yes 24/32 ... yes 3/65 ... yes 91/98 ... yes 91/99 ... yes 91/100 ... yes 91/101 ... yes 91/102 ... yes 91/103 ... yes 4/135 ... yes 3/136 ... yes 3/137 ... yes 3/138 ... yes 3/139 ... yes 4/172 ... yes 3/173 ... yes 3/206 ... yes 4/207 ... yes 3/208 ... yes 130/241 ... yes 18/242 ... yes 20/243 ... yes 233/244 ... yes 3/252 ... yes 3/253 ... yes 3/254 ... yes 4/255 ... yes 235/256 ... yes 130/257 ... yes 7/258 ... yes 130/259 ... yes 4/260 ... yes 3/261 ... yes 3/262 ... yes 3/264 ... yes 130/265 ... yes 243/267 ... yes 244/268 ... yes 130/270 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.5.5) Git version >= 2.21.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1561388298 sudo mv /home/git/.ssh/environment ~/gitlab-check-backup-1561388298 sudo mv /home/git/.ssh/id_rsa ~/gitlab-check-backup-1561388298 sudo mv /home/git/.ssh/id_rsa.pub ~/gitlab-check-backup-1561388298 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 27

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

app/models/postgresql/replication_slot.rb:31 doesn't have checks for nil values