Use rate limiting Redis instance

Use the new Gitlab::Redis::RateLimiting instance (behind feature toggles) for Rack::Attack and Gitlab::ApplicationRateLimiter.

For Rack::Attack, we use an environment variable. This is because Rack::Attack is configured in an initialiser, so a feature flag is of minimal value - it would still need a restart to take effect. In fact, an environment variable is slightly better because it allows us to control when the application boot happens. If we used a feature flag then the application could boot (and pick up the new value) at any point, which wouldn't happen with an environment variable without a production change issue.

We could rewrite the InstrumentedCacheStore to allow changing the store on a per-operation basis, but that adds more risk for what should be a quick migration. It would also add a feature flag check in a very hot code path (rate limiting checks happen multiple times on every request).

For Gitlab::ApplicationRateLimiter, we can and do use a feature flag. The maximum interval for any rate limit in this section is 3 minutes, so we don't mind too much about requests that sneak in the gaps when the feature flag is being changed - especially as feature flags are cached for a minute in process memory anyway, so it would be hard to be much more precise.

Testing

There are a few combinations here and we'll use the same settings for all of them. At admin/application_settings/network, enable the unauthenticated API request limit and set it to 2 per 60 seconds, as this is very convenient for testing. Similarly, set 'Maximum project export requests per minute' to 1.

You will also need a config/redis.rate_limiting.yml file like the below:

development: "unix:/home/smcgivern/gdk/redis/redis.socket?db=4"
test: "unix:/home/smcgivern/gdk/redis/redis.socket?db=14"

gitlab-development-kit!2181 (merged) can generate this, or you can just copy that (with fixed paths) to the correct location.

Rack::Attack

Without any special env vars set:

$ curl -is http://localhost:3000/api/v4/projects 2>/dev/null | head -n 1
HTTP/1.1 200 OK
$ curl -is http://localhost:3000/api/v4/projects 2>/dev/null | head -n 1
HTTP/1.1 200 OK
$ curl -is http://localhost:3000/api/v4/projects 2>/dev/null | head -n 1
HTTP/1.1 429 Too Many Requests

# This shows that we're using the cache Redis (database 2)
$ gdk redis-cli -n 2 keys '*rack::attack*'
1) "cache:gitlab:rack::attack:27215306:throttle_unauthenticated_api:127.0.0.1"
$ gdk redis-cli -n 4 keys '*rack::attack*'
(empty array)

With USE_RATE_LIMITING_STORE_FOR_RACK_ATTACK=1 (for instance, gdk stop rails-web && USE_RATE_LIMITING_STORE_FOR_RACK_ATTACK=1 bundle exec rails s):

$ curl -is http://localhost:3000/api/v4/projects 2>/dev/null | head -n 1
HTTP/1.1 200 OK
$ curl -is http://localhost:3000/api/v4/projects 2>/dev/null | head -n 1
HTTP/1.1 200 OK
$ curl -is http://localhost:3000/api/v4/projects 2>/dev/null | head -n 1
HTTP/1.1 429 Too Many Requests

# This shows that we're using the rate limiting Redis (database 4)
$ gdk redis-cli -n 2 keys '*rack::attack*'
(empty array)
$ gdk redis-cli -n 4 keys '*rack::attack*'
1) "cache:gitlab:rack::attack:27215309:throttle_unauthenticated_api:127.0.0.1"

Gitlab::ApplicationRateLimiter

Try to export the same project twice. With the feature flag off:

$ gdk redis-cli -n 2 keys '*project_export*'
1) "application_rate_limiter:project_export:user:1"
$ gdk redis-cli -n 4 keys '*project_export*'
(empty array)

After bundle exec rails r 'Feature.enable(:use_rate_limiting_store_for_application_rate_limiter)', do the same and you'll see:

$ gdk redis-cli -n 2 keys '*project_export*'
(empty array)
$ gdk redis-cli -n 4 keys '*project_export*'
1) "application_rate_limiter:project_export:user:1"

For gitlab-com/gl-infra/scalability#1247 (closed).

This builds on and targets !70414 (merged).