GeoRepositorySyncWorker fails due to attempt to update `routes` table
If projects do not have a routes entry, the lazy init will cause GeoRepositorySyncWorker to fail hard and prevent any progress from being made:
00:10:07 rails-background-jobs.1 | 2017-06-26T00:10:07.041Z 21518 TID-gp1ukl87w GeoRepositorySyncWorker JID-c71691d7b4541ed6bd61acab INFO: Started Geo repository syncing for 100 project(s)
00:10:07 postgresql.1 | ERROR: cannot execute INSERT in a read-only transaction
00:10:07 postgresql.1 | STATEMENT: INSERT INTO "routes" ("source_id", "source_type", "path", "name", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"
00:10:07 rails-background-jobs.1 | 2017-06-26T00:10:07.071Z 21518 TID-gp1ukl87w GeoRepositorySyncWorker JID-c71691d7b4541ed6bd61acab INFO: fail: 1.041 sec
00:10:07 rails-background-jobs.1 | 2017-06-26T00:10:07.071Z 21518 TID-gp1ukl87w WARN: {"context":"Job raised exception","job":{"queue":"cronjob","class":"GeoRepositorySyncWorker","args":[],"ret
ry":false,"jid":"c71691d7b4541ed6bd61acab","created_at":1498435806.027618,"enqueued_at":1498435806.0278583},"jobstr":"{\"queue\":\"cronjob\",\"class\":\"GeoRepositorySyncWorker\",\"args\":[],\"
retry\":false,\"jid\":\"c71691d7b4541ed6bd61acab\",\"created_at\":1498435806.027618,\"enqueued_at\":1498435806.0278583}"}
00:10:07 rails-background-jobs.1 | 2017-06-26T00:10:07.071Z 21518 TID-gp1ukl87w WARN: ActiveRecord::StatementInvalid: PG::ReadOnlySqlTransaction: ERROR: cannot execute INSERT in a read-only tr
ansaction
00:10:07 rails-background-jobs.1 | : INSERT INTO "routes" ("source_id", "source_type", "path", "name", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"
00:10:07 rails-background-jobs.1 | 2017-06-26T00:10:07.072Z 21518 TID-gp1ukl87w WARN: /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/peek-pg-1.3.0/lib/peek/views/pg.rb:26:in `exec_pre
pared'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/peek-pg-1.3.0/lib/peek/views/pg.rb:26:in `exec_prepared'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:602:in `block in exec_
cache'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_cache'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in `execute_and_cl
ear'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `ex
ec_query'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:192:in `ex
ec_insert'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:108:in `inse
rt'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:14:in `insert'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation.rb:64:in `insert'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/persistence.rb:524:in `_create_record'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_methods/dirty.rb:133:in `_create_record'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:306:in `block in _create_record'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:117:in `call'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:505:in `call'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:778:in `_run_create_callbacks'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:306:in `_create_record'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/timestamp.rb:57:in `_create_record'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/persistence.rb:504:in `create_or_update'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:302:in `block in create_or_update'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:117:in `call'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:505:in `call'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:778:in `_run_save_callbacks'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/callbacks.rb:302:in `create_or_update'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/persistence.rb:120:in `save'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/validations.rb:37:in `save'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_methods/dirty.rb:21:in `save'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:286:in `block (2 levels) in save'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:351:in `block in with_transaction_returning_stat
us'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `bloc
k in transaction'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_t
ransaction'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `tran
saction'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:220:in `transaction'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:348:in `with_transaction_returning_status'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:286:in `block in save'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:301:in `rollback_active_record_state!'
00:10:07 rails-background-jobs.1 | /usr/local/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/transactions.rb:285:in `save'
00:10:07 rails-background-jobs.1 | /home/stanhu/gdk-ee/gitlab/app/models/concerns/routable.rb:148:in `update_route'
00:10:07 rails-background-jobs.1 | /home/stanhu/gdk-ee/gitlab/app/models/concerns/routable.rb:116:in `uncached_full_path'
00:10:07 rails-background-jobs.1 | /home/stanhu/gdk-ee/gitlab/app/models/concerns/routable.rb:107:in `full_path'
00:10:07 rails-background-jobs.1 | /home/stanhu/gdk-ee/gitlab/app/services/geo/repository_sync_service.rb:154:in `log'
00:10:07 rails-background-jobs.1 | /home/stanhu/gdk-ee/gitlab/app/services/geo/repository_sync_service.rb:108:in `try_obtain_lease'
/cc: @dbalexandre, @brodock