Websocket `ws-xx` servers filling up with logs from revoked users
Websocket ws-xx
servers filling up with logs
$ ssh deployer@ws-06
root@ws-06:/var/log$ du --max-depth=1 | sort -n
4 ./dist-upgrade
4 ./gitter
4 ./landscape
12 ./nginx
20 ./apt
52 ./exim4
72 ./deploy
72 ./unattended-upgrades
316 ./datadog
964 ./cron-monitor
+ 6821376 ./upstart <------------ 6.8GB
+ 7737396 ./td-agent <------------ 7.7GB
14576616
What do the logs contain?
Upstart
The logs contain a bunch of people trying to reconnect with revoked tokens.
I think this is caused by the gitter-realtime-client
usage within the desktop app (not within the webapp
which has already signed them out). We use the gitter-realtime-client
separately in the desktop app to update the task bar and tray menu with unreads. The realtime client will handle "Reinitialisation of state after client disconnections" and doesn't have anything in place to stop reconnecting after a 401
/403
like we have in the webapp
.
These people haven't noticed that Gitter has revoked them because they just leave it running in the background.
This log is being removed in https://gitlab.com/gitlab-org/gitter/webapp/merge_requests/1185 as a temporary fix
For the future, we should fix the gitter-realtime-client
in the desktop app to not reconnect after 401
/403
, https://gitlab.com/gitlab-org/gitter/desktop/issues/253
$ ssh deployer@ws-06
$ sudo su
$ less +G /var/log/upstart/gitter-websockets-1.log.1
...
2018-06-16T22:26:26.807Z - error: bayeux: extension authenticator[incoming] failed: 401::Unauthorized message=Unauthorized, status=401, channel=/meta/handshake, token=xxx, ip=10.0.10.13, connection=close
2018-06-16T22:26:26.848Z - warn: Token can not be accepted (client has been revoked): token=xxx, clientId=xxx
2018-06-16T22:26:26.848Z - error: bayeux: extension authenticator[incoming] failed: 401::Unauthorized message=Unauthorized, status=401, channel=/meta/handshake, token=xxx, ip=10.0.10.13, connection=close
2018-06-16T22:26:26.944Z - warn: Token can not be accepted (client has been revoked): token=xxx, clientId=xxx
2018-06-16T22:26:26.944Z - error: bayeux: extension authenticator[incoming] failed: 401::Unauthorized message=Unauthorized, status=401, channel=/meta/handshake, token=xxx, ip=10.0.10.13, connection=close
2018-06-16T22:26:27.011Z - warn: Token can not be accepted (client has been revoked): token=xxx, clientId=xxx
td-agent
/fluentd
TODO: idk
$ ssh deployer@ws-06
$ sudo su
$ less +G /var/log/td-agent/td-agent.log
...
2018-06-16 22:29:42 +0000 [error]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2018-06-16 22:29:42 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-1.1.0/lib/fluent/plugin/buffer.rb:265:in `write'" tag="udp"
2018-06-16 22:29:42 +0000 [warn]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [error]: #0 unexpected error in processing UDP data error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data"
2018-06-16 22:29:42 +0000 [error]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2018-06-16 22:29:42 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-1.1.0/lib/fluent/plugin/buffer.rb:265:in `write'" tag="udp"
2018-06-16 22:29:42 +0000 [warn]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [error]: #0 unexpected error in processing UDP data error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data"