Add LabKit Rate_Limit_Block lines to rate limit troubleshooting

### Quick Links

- [Metrics: Rate Limiting Overview dashboard](https://dashboards.gitlab.net/d/rate-limiting-rate-limiting_overview/rate-limiting3a-rate-limiting3a-overview?orgId=1)

- [Logs: RackAttack](https://log.gprd.gitlab.net/app/discover#/view/0026cc97-6b9a-445a-a364-7197e04053a2?_g=())

- [Logs: RackAttack and LabKit rate limiter](https://log.gprd.gitlab.net/app/discover#/view/0026cc97-6b9a-445a-a364-7197e04053a2?_g=())

- [Logs: ApplicationRateLimiter](https://log.gprd.gitlab.net/app/discover#/view/2d2cf10e-b22a-4c07-bbda-45bb665c31ee?_g=())

- [Logs: Rate Limit Dashboard](https://log.gprd.gitlab.net/app/r/s/AJDZC)

- Throttle using `json.matched`

- Path using `json.path`

### LabKit rate limiter

When the LabKit rate limiter blocks a request, the response also contains `RateLimit-*` response headers.

GitLab does not log a `Rack_Attack` line for the request.

Instead, it logs a `Rate_Limit_Block` line in the Rails logs, with `json.subcomponent: auth_json`.

LabKit logs only some blocked requests: in each rate limit window, the first request over the limit of a rule, and then the requests at each power of two times the limit (2, 4, 8, and so on).

Most blocked requests have no log line, so a Workhorse `correlation_id` usually has no match.

The [RackAttack logs](https://log.gprd.gitlab.net/app/discover#/view/0026cc97-6b9a-445a-a364-7197e04053a2?_g=()) show both `Rack_Attack` and `Rate_Limit_Block` lines.

Filter on `json.message: Rate_Limit_Block`, then filter by:

- IP address using `json.remote_ip`

- The client that the rule counted using `json.counted_by.ip`, or `json.counted_by.requester_type` and `json.counted_by.requester_id` for a rule that counts by authenticated requester (`requester_type` is `user` or `deploy_token`)

- Throttle using `json.matched`, or the LabKit rule using `json.rule`

- Path using `json.path`

`json.remote_ip`, `json.matched`, and `json.path` have the same names as in the RackAttack logs, so one filter finds both lines.

`json.path` has no query string.

For all the fields, see [Find the source of a 429](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/rate-limiting/labkit-cohort-flags.md#find-the-source-of-a-429) in the runbooks.

### ApplicationRateLimiter

You can filter the [ApplicationRateLimiter logs](https://log.gprd.gitlab.net/app/discover#/view/2d2cf10e-b22a-4c07-bbda-45bb665c31ee?_g=()) by:

### Workhorse

If you have not found the request in Cloudflare, RackAttack, or ApplicationRateLimiter,

If you have not found the request in Cloudflare, RackAttack, the LabKit rate limiter, or ApplicationRateLimiter,

then you can search for rate limited responses in the [Workhorse logs](https://log.gprd.gitlab.net/app/discover#/view/7b6dc396-5b27-4e86-b150-72b476255faf?_g=()) by:

- IP using `json.remote_ip`

It's common to see multiple failed authentication requests (401) trigger a Rack Attack block which causes a 403 Forbidden message. We block IP addresses if we receive [300 failed requests from a single IP in a one-minute period](https://docs.gitlab.com/user/gitlab_com/rate_limits/#git-and-container-registry-failed-authentication-ban). It's worth noting that by default, Git operations are first tried unauthenticated so it's expected to see two 401 responses for every Git operation.

Rack Attack can also *throttle* traffic. If that is the case, this is recognizable by the HTTP 429 response code. The preferred solution to this is to have the user make fewer requests. Please refer to our [Bypass Policy](ip-blocks.md#bypass-policy) for more information.

Rack Attack, or the LabKit rate limiter, can also *throttle* traffic. If that is the case, this is recognizable by the HTTP 429 response code. The preferred solution to this is to have the user make fewer requests. Please refer to our [Bypass Policy](ip-blocks.md#bypass-policy) for more information.

The LabKit rate limiter can return the 429s that Rack Attack throttles used to return.

GitLab then writes no `Rack_Attack` line for them.

If a `Rack_Attack` filter shows no result, also try the `json.message` filter `Rate_Limit_Block`.

LabKit logs only some blocked requests: in each rate limit window, the first request over the limit of a rule, and then the requests at each power of two times the limit (2, 4, 8, and so on).

If you find no line for a specific request, it can still have been rate limited.

Remove the `json.meta.remote_ip` filter from the [search condition](#search-condition): neither `Rack_Attack` nor `Rate_Limit_Block` lines have that field.

Filter the results by `json.remote_ip` instead, or by `json.counted_by.requester_type` and `json.counted_by.requester_id` for a rule that counts by authenticated requester, such as a user or a deploy token.

For more information, see [LabKit rate limiter](/handbook/engineering/infrastructure-platforms/rate-limiting/troubleshooting/#labkit-rate-limiter).

### Checking the Application Rate Limiter

- `json.status` - Outputs the HTTP status code that was returned for the request. We're usually looking for `401` (Unauthorized) and/or `403` (Forbidden).

- `json.path` - The path on GitLab.com that was accessed by the request or the API endpoint that was hit.

- `json.method` - Can be either `GET`, `POST`, `PUT`, `PATCH`, or `DELETE`. The first three are the most common.

- `json.env` - Can be `blocklist`, `throttle` or `track`. `track` is used for diagnostics when changing rate limiting settings and does not affect users: from a support perspective, ignore `track`. `blocklist` happens in reaction to too many failed authentication attempts, for example with automated Git HTTP traffic. `throttle` means a user or IP is making too many requests per minute.

- `json.env` - Can be `blocklist`, `throttle` or `track`. `track` is used for diagnostics when changing rate limiting settings and does not affect users: from a support perspective, ignore `track`. `blocklist` happens in reaction to too many failed authentication attempts, for example with automated Git HTTP traffic. `throttle` means a user or IP is making too many requests per minute. `Rate_Limit_Block` lines have no `json.env` field, so a `json.env` filter hides them.

#### Secondary

- `json.controller` - Gives you a clue as to what part of GitLab.com was being accessed by a particular request.

- `json.params` - Shows what user made the request, what action was taken, and on what resource it was taken on. This field shows what repository was targeted for requests to the container registry.

- `json.matched` - This is the name of the Rack Attack rule used to limit this request (when `json.env` is `throttle`). This can help you find out which of the [current limits](https://docs.gitlab.com/user/gitlab_com/rate_limits/#current-rate-limits) the request surpassed.

- `json.matched` - This is the name of the Rack Attack rule used to limit this request (when `json.env` is `throttle`, or on a `Rate_Limit_Block` line). This can help you find out which of the [current limits](https://docs.gitlab.com/user/gitlab_com/rate_limits/#current-rate-limits) the request surpassed.

## Common Causes