Skip to content

Add a redis rate limiter for agentk connections

Hordur Freyr Yngvason requested to merge add-redis-based-rate-limiter into master

First part of #25 (closed)

This adds redis configuration for kas and a redis-based rate limiter for new agent connections per minute. Redis, and by extension the limiter, start out as optional because downstream consumers (omnibus, helm, gdk) do not have redis configured for kas.

Manual QA

Configured the GDK kas with

redis:
  url: unix:///Users/hfyngvason/src/gdk-ee/redis/redis.socket

agent:
  limits:
    connections_per_token_per_minute: 1

and set up a manifest project with an agentk. Then, when scaling the agentk up, we get warning logs of the form

2020-10-17_03:11:31.67300 gitlab-k8s-agent      : time="2020-10-16T23:11:31-04:00" level=warning msg="redis.TokenLimiter: Rate limit exceeded" count=1 key="kas:agent_limits:\xda\xce\x16Or\xee\xfa8\xe6\xaft\x1eoʊ{\xdd^%\xe5\xfb\x04\xe2Z%Jť\xc3t\xa1\x03:11" limit=1
2020-10-17_03:11:37.58771 gitlab-k8s-agent      : time="2020-10-16T23:11:37-04:00" level=warning msg="redis.TokenLimiter: Rate limit exceeded" count=1 key="kas:agent_limits:\xda\xce\x16Or\xee\xfa8\xe6\xaft\x1eoʊ{\xdd^%\xe5\xfb\x04\xe2Z%Jť\xc3t\xa1\x03:11" limit=1

TODO

  • Test manually with a real redis
  • Encapsulate redigo, use interfaces for connection pool
  • Set sensible defaults
  • Add Redis Sentinel support (added but not tested)
  • [-] Improve error handling This will be a follow up
  • Add tests
Edited by Hordur Freyr Yngvason

Merge request reports