Skip to content

Draft: feat(errortracking): Add ability to perform client-side buffering for writes

Ankit Bhatnagar requested to merge abhatnagar/errortracking-buffered-writes into main

The current implementation leverages db.conn.AsyncInsert which acquires a connection per invocation. The connection acquisition happens against a pool of connection which can be tuned via db.MaxOpenConns && db.MaxIdleConns.

While this works well, it does not scale beyond the 1000 requests/second benchmark as seen here: !1638 (comment 1013614603)

Bumping up the size of the connection pool is limited and cannot be increased infinitely. Assuming we want to run the errortracking ingestion endpoint at requests rates larger than ~1000, it'd be prudent to client-side buffer these writes and dump them into CH via a PrepareBatch then Send call.

The current implementation attempts to do this, allowing us to tweak behaviour on two different accounts:

  • By limiting the size of the buffer i.e. flush to CH every n buffered writes.
  • By limiting the flush interval i.e. flush to CH every n seconds (or whatever time.Duration) via a time.Ticker.

(nit) It also segregates read, write and table specific implementations into their own go files.

The only downside to this might be the lack of feedback immediately per request, esp. when running at higher traffic, but that should be intended behaviour. That said, should such be the case, the code can be changed to add error/feedback per write.

The implementation is also done in a way to extract it into a common shared library for re-use into other CH-backed storage implementations, e.g. metrics.

Ref: #1672

Edited by Ankit Bhatnagar

Merge request reports