Simplify tunnel registry
We've been optimizing the tunnel registry by using sharding over the agent IDs. So each registry stripe maintains a bunch of tunnels for some agent IDs that are protected by a shared mutex within that stripe. The tunnel registration and unregistration are locked by that stripe-local mutex - including IO to Redis. Given that we've recently lowered the number of KAS replicas which impacted the amount of agents within each stripe we've also noted some increased mutex contention on those stripes.
The entire implementation today is also rather complicated.
The proposal of this issue is to simplify the tunnel registry by:
- Getting rid of sharding.
- Introducing a single worker accepting IO requests.
- Registry sends serialized commands to the worker for it to perform the IO operations asynchronously. We never have to wait for the result.
- Worker could further be optimized by batching the Redis commands.