Skip to content

Exclude "/api/v4/geo/proxy" in Rack Attack

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

If you enabled rate limiting in the secondary node when using GitLab Geo, one of the Geo related path (/api/v4/geo/proxy) is adding to the unauthenticated rate limit.

Proxy endpoint implementation can be found here: https://gitlab.com/gitlab-org/gitlab/-/blob/99b08ac9f1fe7791806684a2177b4ea3e6503a71/lib/api/geo.rb#L23-32

Steps to reproduce

  1. Setup GitLab Geo.
  2. Configure Rate limiting.
  3. Notice that in the logs of the secondary node, we can see /api/v4/geo/proxy adding to the unauthenticated rate limit.

What is the current bug behavior?

/api/v4/geo/proxy is not excluded in the rate limit.

What is the expected correct behavior?

/api/v4/geo/proxy should be excluded in the rate limit.

Relevant logs and/or screenshots

{
    "component":"gitlab",
    "subcomponent":"auth_json",
    "level":"error",
    "severity":"ERROR",
    "time":"2023-03-06T16:30:16.731Z",
    "correlation_id":"d16a02fa-2049-4be8-b1d5-bce5164f990f",
    "message":"Rack_Attack",
    "env":"track",
    "remote_ip":"127.0.0.1",
    "request_method":"GET",
    "path":"/api/v4/geo/proxy",
    "matched":"throttle_unauthenticated_api",
    "redis_calls":2,
    "redis_duration_s":0.019386,
    "redis_read_bytes":8,
    "redis_write_bytes":152,
    "redis_rate_limiting_calls":2,
    "redis_rate_limiting_duration_s":0.019386,
    "redis_rate_limiting_read_bytes":8,
    "redis_rate_limiting_write_bytes":152,
    "db_count":0,
    "db_write_count":0,
    "db_cached_count":0,
    "db_replica_count":0,
    "db_primary_count":0,
    "db_main_count":0,
    "db_geo_count":0,
    "db_main_replica_count":0,
    "db_replica_cached_count":0,
    "db_primary_cached_count":0,
    "db_main_cached_count":0,
    "db_geo_cached_count":0,
    "db_main_replica_cached_count":0,
    "db_replica_wal_count":0,
    "db_primary_wal_count":0,
    "db_main_wal_count":0,
    "db_geo_wal_count":0,
    "db_main_replica_wal_count":0,
    "db_replica_wal_cached_count":0,
    "db_primary_wal_cached_count":0,
    "db_main_wal_cached_count":0,
    "db_geo_wal_cached_count":0,
    "db_main_replica_wal_cached_count":0,
    "db_replica_duration_s":0.0,
    "db_primary_duration_s":0.0,
    "db_main_duration_s":0.0,
    "db_geo_duration_s":0.0,
    "db_main_replica_duration_s":0.0,
    "rack_attack_redis_count":1,
    "rack_attack_redis_duration_s":0.019716963171958923,
    "cpu_s":0.006833,
    "mem_objects":2551,
    "mem_bytes":331616,
    "mem_mallocs":848,
    "mem_total_bytes":433656,
    "pid":39,
    "worker_id":"puma_0",
    "rate_limiting_gates":[
    ]
}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible workarounds

Possible workaround 1

Increase rate limit until it doesn't trigger it.

Possible workaround 2

Somehow add a header to Workhorse requests which originate from Workhorse and not from an external client https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html#use-an-http-header-to-bypass-rate-limiting, though be careful not to allow all Workhorse requests indiscriminately since user requests come through Workhorse as well.

Related issue

If we had #25597 then maybe we could allowlist remote_ip 127.0.0.1

Possible fixes

Possible fix 1

In https://gitlab.com/gitlab-org/gitlab/-/blob/c04553e111a1c48767182e60d3739ffe9510f238/ee/lib/ee/gitlab/rack_attack/request.rb#L14, return true if the path is api/v4/geo/proxy.

Possible fix 2

In https://gitlab.com/gitlab-org/gitlab/-/blob/c04553e111a1c48767182e60d3739ffe9510f238/ee/lib/ee/gitlab/rack_attack/request.rb#L14, return true if the IP is 127.0.0.1. Is it safe to assume that remote_ip is 127.0.0.1 only when the request originated from an internal service such as Workhorse or GitLab Shell?

Edited by 🤖 GitLab Bot 🤖