gitlab-monitor causes MySQL to stop allowing connections
I've been doing some testing over the last week with GitLab 9.2 and seeing a lot of issues with connections to MySQL that I've not seen before. On a freshly built machine talking to a MySQL backend using the MySQL client 5.5.56 that was updated in this release I am seeing a lot of the following error in the unicorn_stderr.log
file:
I, [2017-05-30T14:15:30.872639 #4938] INFO -- : Refreshing Gem list
I, [2017-05-30T14:16:04.285801 #4938] INFO -- : listening on addr=127.0.0.1:8080 fd=14
I, [2017-05-30T14:16:04.286078 #4938] INFO -- : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I, [2017-05-30T14:16:04.286198 #4938] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=15
E, [2017-05-30T14:16:04.288624 #4938] ERROR -- : Host '10.51.28.62' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' (Mysql2::Error)
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/mysql2-0.3.20/lib/mysql2/client.rb:70:in `connect'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/mysql2-0.3.20/lib/mysql2/client.rb:70:in `initialize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `new'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `mysql2_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/opt/gitlab/embedded/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_handling.rb:87:in `connection'
/var/opt/gitlab/gitlab-rails/etc/unicorn.rb:32:in `block in reload'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:502:in `spawn_missing_workers'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:132:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
/opt/gitlab/embedded/bin/unicorn:22:in `load'
/opt/gitlab/embedded/bin/unicorn:22:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
If I flush the hosts it works for a little while but will ultimately break again.
I've just built a new version of the package with 5.5.54 of the MySQL client and so far have not seen the error but will do some more testing to see if it appears.
I've seen the error on a couple of different installations now, pointing to different MySQL servers running different versions of MySQL so I don't think there is anything particular in the MySQL servers that would be causing it but am obviously wary of the client update at the moment.