Enable Action Cable for GitLab.com

This issue will track the work for enabling the Action Cable feature on GitLab.com

Copied from &228 (comment 414329841)

@engwan

what ways do we have to monitor this with logs or metrics after it is enabled?

We have Rails logs for every connection and subscription made and we also have Prometheus metrics which shows the status of the ActionCable thread pool.

When this is enabled, this won't immediately see any load though because there is a feature flag on the client side. So we can roll this out gradually.

I think one of the other concerns we must decide on is the Redis instance. Should we create a separate Redis instance / fleet for ActionCable? Or reuse the SharedState? (I believe our charts already support specifying which Redis instance to use for ActionCable)

ActionCable uses Redis only for PUBSUB and does not persist anything.

Copied from &228 (comment 414346096)

@smcgivern

I'd be OK with going with the persistent Redis at least while this is still controlled by a feature flag. If we see problems we can disable the flag and spin up a separate Redis, but I hope that's unlikely.

Copied from &228 (comment 335844114)

@nick.thomas

we just merged a change to workhorse to support proxying the /-/cable endpoint, so the traffic would go through workhorse, yes.

I'd propose that we set the configuration in omnibus + CNG so workhorse proxies these websockets to the same place as the rest of the rails requests by default. This feature is going to be a pain if it's off-by-default.

HA can segment websocket traffic, either at the haproxy or workhorse level, to go to a dedicated set of rails processes.

While introducing websockets for online terminals, there was a lot of worry about the infrastructure costs of maintaining long-lived, stateful connections. Online terminals managed to sidestep that by virtue of the state being unavoidably ephemeral to a single TCP connection; presumably the same isn't true of actioncable?

Edited by John Jarvis