Skip to content

Add configurable setting for "Project Jobs API endpoint"

Ben King requested to merge benjaminking-jobs-index-setting into master

What does this MR do and why?

Raised from this issue: #395702 (closed)

This is an initial proof of concept MR to allow the Project Jobs API endpoint rate limit to be configurable, instead of the hard-coded value of 600 calls per minute.

For initial verification purposes, the UI setting has been placed under User and IP rate limits, but can likely be moved to a more suitable location.

A database migration has been added to define the rate limit setting, which can then be configured via UI or via Rails Console with:

::Gitlab::CurrentSettings.update!(project_jobs_api_rate_limit: <int>)

Screenshots or screen recordings

Proof of Concept of the setting added to the UI:

image

How to set up and validate locally

  1. Deploy a GDK instance.
  2. Go to Admin Area > Settings > Network
  3. Under User and IP rate limits, confirm that the value for Maximum authenticated requests to project/:id/jobs per minute is set to the default of 600.
  4. Reduce this value to a small value for testing, such as 10, then select Save changes.
  5. Open a Rails console and confirm the setting has been saved to the database with: ::Gitlab::CurrentSettings.project_jobs_api_rate_limit
  6. Locate a project, and copy the Project ID for reference.
  7. Create a Personal Access Token, then save this value for reference.
  8. Run the following bash script against the GDK instance to create calls to the rate-limited endpoint. It should change from 200 responses to 429 to indicate the API is being rate-limited:
TOKEN=<token>
ID=<id>
URL="<url>"
for i in {1..20}
do
  echo -n "Attempt $i: "
  curl -i --header "PRIVATE-TOKEN: $TOKEN" "$URL/api/v4/projects/$ID/jobs" -o /dev/null -w '%{http_code}\n' -s
done

Example Output:

Attempt 1: 200
Attempt 2: 200
Attempt 3: 200
Attempt 4: 200
Attempt 5: 200
Attempt 6: 200
Attempt 7: 200
Attempt 8: 200
Attempt 9: 200
Attempt 10: 200
Attempt 11: 429
Attempt 12: 429
Attempt 13: 429
....

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Ben King

Merge request reports