Websocket `ws-xx` servers filling up with logs from revoked users

Websocket ws-xx servers filling up with logs

$ ssh deployer@ws-06
root@ws-06:/var/log$ du --max-depth=1 | sort -n
4       ./dist-upgrade
4       ./gitter
4       ./landscape
12      ./nginx
20      ./apt
52      ./exim4
72      ./deploy
72      ./unattended-upgrades
316     ./datadog
964     ./cron-monitor
+ 6821376 ./upstart <------------ 6.8GB
+ 7737396 ./td-agent <------------ 7.7GB
14576616

What do the logs contain?

Upstart

The logs contain a bunch of people trying to reconnect with revoked tokens.

I think this is caused by the gitter-realtime-client usage within the desktop app (not within the webapp which has already signed them out). We use the gitter-realtime-client separately in the desktop app to update the task bar and tray menu with unreads. The realtime client will handle "Reinitialisation of state after client disconnections" and doesn't have anything in place to stop reconnecting after a 401/403 like we have in the webapp.

These people haven't noticed that Gitter has revoked them because they just leave it running in the background.

This log is being removed in https://gitlab.com/gitlab-org/gitter/webapp/merge_requests/1185 as a temporary fix

For the future, we should fix the gitter-realtime-client in the desktop app to not reconnect after 401/403, https://gitlab.com/gitlab-org/gitter/desktop/issues/253

$ ssh deployer@ws-06
$ sudo su
$ less +G /var/log/upstart/gitter-websockets-1.log.1
...
2018-06-16T22:26:26.807Z - error: bayeux: extension authenticator[incoming] failed: 401::Unauthorized message=Unauthorized, status=401, channel=/meta/handshake, token=xxx, ip=10.0.10.13, connection=close
2018-06-16T22:26:26.848Z - warn: Token can not be accepted (client has been revoked):  token=xxx, clientId=xxx
2018-06-16T22:26:26.848Z - error: bayeux: extension authenticator[incoming] failed: 401::Unauthorized message=Unauthorized, status=401, channel=/meta/handshake, token=xxx, ip=10.0.10.13, connection=close
2018-06-16T22:26:26.944Z - warn: Token can not be accepted (client has been revoked):  token=xxx, clientId=xxx
2018-06-16T22:26:26.944Z - error: bayeux: extension authenticator[incoming] failed: 401::Unauthorized message=Unauthorized, status=401, channel=/meta/handshake, token=xxx, ip=10.0.10.13, connection=close
2018-06-16T22:26:27.011Z - warn: Token can not be accepted (client has been revoked):  token=xxx, clientId=xxx

td-agent/fluentd

TODO: idk

$ ssh deployer@ws-06
$ sudo su
$ less +G /var/log/td-agent/td-agent.log
...
2018-06-16 22:29:42 +0000 [error]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2018-06-16 22:29:42 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-1.1.0/lib/fluent/plugin/buffer.rb:265:in `write'" tag="udp"
  2018-06-16 22:29:42 +0000 [warn]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [error]: #0 unexpected error in processing UDP data error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data"
  2018-06-16 22:29:42 +0000 [error]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2018-06-16 22:29:42 +0000 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-1.1.0/lib/fluent/plugin/buffer.rb:265:in `write'" tag="udp"
  2018-06-16 22:29:42 +0000 [warn]: #0 suppressed same stacktrace
2018-06-16 22:29:42 +0000 [error]: #0 unexpected error in processing UDP data error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data"
Edited by Eric Eastwood