Fluentd throw undefined method `synchronize' for nil:NilClass when config is reloaded
Summary
When configuration is reloaded we can find an error in fleuntd:
undefined method `synchronize' for nil:NilClass
This error is followed by a:
failed to reload config: Address already in use
Current Behaviour
The reload is not correctly handled
Steps to reproduce
- Create a distirbuted logs and wait for its Pod to be Ready
- Create a cluster that point to previously created distributed logs
- Sometimes in the
fluentd
container of the distributed logs Pod the error appears
Expected Behaviour
No error appears in the fluentd
container logs
Possible Solution
None apart from waiting this issue is solved in fluentd and see if that fix our problem.
Environment
- StackGres version: 1.0.0-alpha2-SNAPSHOT
- Kubernetes version: 1.16.15
- Cloud provider or hardware configuration: 32 cores / 128 GB RAM
Relevant logs and/or screenshots
2021-04-23T11:17:04.633378288Z 2021-04-23 11:17:04 +0000 [error]: #1 unexpected error error_class=NoMethodError error="undefined method `synchronize' for nil:NilClass"
2021-04-23T11:17:04.633383428Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin_helper/event_loop.rb:39:in `event_loop_attach'
2021-04-23T11:17:04.633395741Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin_helper/server.rb:208:in `server_attach'
2021-04-23T11:17:04.633441217Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin_helper/server.rb:108:in `server_create_connection'
2021-04-23T11:17:04.633739438Z 2021-04-23 11:17:04 +0000 [info]: #1 Selecting 'log_patroni' table
2021-04-23T11:17:04.633747583Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin/in_forward.rb:172:in `start'
2021-04-23T11:17:04.634707108Z 2021-04-23 11:17:04 +0000 [info]: #1 Selecting 'log_postgres' table
2021-04-23T11:17:04.634723920Z 2021-04-23 11:17:04 +0000 [info]: #1 listening port port=12226 bind="127.0.0.1"
2021-04-23T11:17:04.635607191Z 2021-04-23 11:17:04 +0000 [warn]: #1 thread exited by unexpected error plugin=Fluent::Plugin::ForwardInput title=:event_loop error_class=NoMethodError error="undefined method `synchronize' for nil:NilClass"
2021-04-23T11:17:04.635635635Z #<Thread:0x00005608c6be2c68@event_loop@/usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin_helper/thread.rb:70 run> terminated with exception (report_on_exception is true):
2021-04-23T11:17:04.635643480Z /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin_helper/event_loop.rb:39:in `event_loop_attach': undefined method `synchronize' for nil:NilClass (NoMethodError)
2021-04-23T11:17:04.635652016Z from /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin_helper/event_loop.rb:91:in `block in start'
2021-04-23T11:17:04.635655873Z from /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-04-23T11:17:04.635920050Z 2021-04-23 11:17:04 +0000 [error]: #1 failed to reload config: Address already in use - bind(2) for 127.0.0.1:12226
2021-04-23T11:17:04.640168211Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/root_agent.rb:200:in `block in start'
2021-04-23T11:17:04.640223445Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/root_agent.rb:189:in `block (2 levels) in lifecycle'
2021-04-23T11:17:04.640228956Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/root_agent.rb:188:in `each'
2021-04-23T11:17:04.640232292Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/root_agent.rb:188:in `block in lifecycle'
2021-04-23T11:17:04.640234777Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/root_agent.rb:175:in `each'
2021-04-23T11:17:04.640237612Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/root_agent.rb:175:in `lifecycle'
2021-04-23T11:17:04.640239856Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/root_agent.rb:199:in `start'
2021-04-23T11:17:04.640262248Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/engine.rb:248:in `start'
2021-04-23T11:17:04.640267087Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/engine.rb:147:in `run'
2021-04-23T11:17:04.640279761Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/supervisor.rb:700:in `block in run_worker'
2021-04-23T11:17:04.640282937Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/supervisor.rb:951:in `main_process'
2021-04-23T11:17:04.640285071Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/supervisor.rb:691:in `run_worker'
2021-04-23T11:17:04.640287516Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/lib/fluent/command/fluentd.rb:365:in `<top (required)>'
2021-04-23T11:17:04.640290912Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
2021-04-23T11:17:04.640301813Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require'
2021-04-23T11:17:04.640305239Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/share/gems/gems/fluentd-1.12.1/bin/fluentd:8:in `<top (required)>'
2021-04-23T11:17:04.640309026Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/local/bin/fluentd:23:in `load'
2021-04-23T11:17:04.640312553Z 2021-04-23 11:17:04 +0000 [error]: #1 /usr/local/bin/fluentd:23:in `<main>'
Edited by Matteo Melli