Allow ClickHouse connection pool to operate independently from errortracking API autoscaling
Proposal
Use shared connection pool for connectivity between errortracking API and Clickhouse.
Currently the connection pool is managed directly by the errortracking API, however this means that auto-scaling of the errortracking pods will linearly scale DB connections, leading to exhaustion. Instead we should enable a shared connection pool so errortracking API can scale independently of the clickhouse max connections.
One way of doing this could be enablement of CHProxy as a middleware. We currently have some progress towards enabling CHProxy but I don't believe we are yet using it.
Background
The following discussion from gitlab-com/gl-infra/readiness!112 (merged) should be addressed:
-
@mwasilewski-gitlab started a discussion: (+2 comments) Looking at the CHProxy repo it doesn't seem to have a connection pool or a query queue as features. Is that correct? Do we know at this point if these will be provided anywhere in our system?
I see there's some rate limiting capability in CHProxy, is it going to be used? Or are rate limits going to be implemented somewhere else?