feat(rate_limit): add BlockSummary to log block counts per endpoint and rule

Description

Adds Labkit::RateLimit::BlockSummary. It counts blocks per (endpoint_id, rule) in one process and logs one line per pair every interval. A line per block grows with an attack. These lines grow only with the number of distinct pairs, and never past max_pairs + 1 per process per interval.

summary = Labkit::RateLimit::BlockSummary.new
Thread.new { summary.call }
summary.add(endpoint_id: "GET /api/:version/projects", rule: "unauthenticated_api", feature_category: "source_code_management")
# on shutdown
summary.stop
{"message": "rate_limit_block_summary", "endpoint_id": "GET /api/:version/projects", "rate_limit_rule": "unauthenticated_api", "block_count": 412, "feature_category": "source_code_management"}
  • The key is the pair. The first non-nil feature_category passed for a pair is the one logged, so a category never adds a line.
  • Caps at max_pairs pairs per interval (default 10). The rest fold into one other line with no feature_category, so memory and lines stay bounded during an attack.
  • The first flush waits a random part of the interval, so workers booted together do not flush at once.
  • stop signals the thread running #call, which runs a final flush. With no thread in #call, nothing is flushed. A process that exits without stop loses up to one interval of counts.
  • A logging failure is logged at WARN with message: "rate_limit_error" and dropped_pairs, and never raises.
  • new raises ArgumentError on a non-positive interval or max_pairs.
  • The caller owns the thread and decides what endpoint_id means. LabKit does not resolve it.

The log line is pinned in labkit-spec first: gitlab-org/quality/tooling/labkit-spec!50 (merged). This MR is set to merge after that one.

Testing

  • spec/labkit/rate_limit: 469 examples, 0 failures. RuboCop clean.
  • A gitlab-rails version of the same tally ran end to end on a local Caproni cluster: 5 blocked requests to /explore/projects gave one line with count: 5 (the field was named count before the rename to block_count).

gitlab-org/gitlab#638007

Checklist

  • labkit-spec MR merged
  • Teammate review
  • Maintainer review
Edited by Nidhey Indurkar

Merge request reports

Loading
Loading