Delete environments in batches with a per-user rows limit

What does this MR do and why?

Context

Deleting an environment cascades over all its deployments and job links in one transaction. A large environment exceeds the statement timeout, rolls back, and fails the same way on every retry. Nothing bounds how many rows one user deletes. The figures and the agreed approach are in https://gitlab.com/gitlab-org/gitlab/-/work_items/638251#note_4183087864.

This MR

  • Environments::DestroyService deletes the environment's job links, then its merge request links, then its deployments, in batches of 500 that each commit on their own. Before each batch it checks that the environment is still stopped. Then it locks the environment, checks again, and destroys it.
  • Each request stops at a row cap or a time budget. A request that stops early returns an error asking the caller to retry and keeps what it deleted, so retries finish the environment. The cap is the lower of 20,000 rows and the enforced rate limit, and a request stops before the batch that would pass it.
  • This covers REST DELETE /projects/:id/environments/:environment_id, GraphQL environmentDelete and the hourly auto-delete. With the flag on, REST returns 409 Conflict when the request stopped early, 429 Too Many Requests when the user is over the rate limit, and 403 Forbidden when a deployment made the environment available again between batches. With the flag off, REST runs today's code unchanged. GraphQL returns the same messages in errors.
  • The rows each batch deletes count toward a new per-user rate limit, environment_delete_rows_limit_per_user (default 20,000 per minute, 0 disables), on Admin > Settings > Network and in the application settings API. An over-limit batch stops the next batch, so a request that deleted the last rows still deletes the environment. The auto-delete has no user, so the rate limit doesn't apply to it. Each auto-delete job stops starting environments after 4 minutes and logs the environments it leaves for the next run.
  • The key starts in log mode: requests over the limit are logged until rate_limit_environment_delete_rows is enabled, and then refused.
  • A batch can return fewer rows than it selected while a concurrent request deletes the same environment, so the job-link loop ends only when no job links remain. That check is one index lookup, normally once per request, and the time budget bounds the loop.
  • FastDestroyAll#fast_destroy_all now returns the number of deleted rows. No existing caller uses its return value.
  • Docs: a new rate limit page under doc/rate_limits/api/, the environments API, the settings API and the GraphQL mutation description. The Changelog entry comes with the flag-removal MR.

The batches run in the web request rather than a worker, although the batching guidelines prefer Sidekiq. A successful response then still means the environment is gone, and the cap, time budget and rate limit bound each request.

The environments UI deletes through the REST API. !269315 (merged) updated its error message for the retry and rate-limit errors.

Feature Flag: environment_delete_in_batches

Feature Flag: rate_limit_environment_delete_rows

References

Screenshots or screen recordings

The new Environments API rate limits block on Admin > Settings > Network. Before (master) and After (this branch) are attached at the end of this description.

How to set up and validate locally

  1. In a project you maintain with no merged merge requests, create three stopped environments with a few deployments each through the Deployments API (no jobs, so no job links), and note their IDs.
  2. In a Rails console, enable batched deletes and set a small limit:
    Feature.enable(:environment_delete_in_batches)
    ApplicationSetting.current.update!(environment_delete_rows_limit_per_user: 1)
  3. Wait a minute for the settings cache, then delete the first environment with a personal access token (api scope):
    curl --include --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \
      "http://gdk.test:3000/api/v4/projects/<id>/environments/<environment_id>"
    It returns 204, and log/api_json.log shows rate_limit_state with limit_environment_delete_rows_by_user_log:log.
  4. Run Feature.enable(:rate_limit_environment_delete_rows), wait a minute after the first delete, and delete the second and third environments in turn. The second returns 204, because a request that deletes the last rows still finishes. The third returns 429 with Retry-After: 60 and deletes nothing.
  5. Clean up: Feature.remove both flags and reset the setting to 20000.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

before after

Edited by Leaminn Ma

Merge request reports

Loading
Loading