Meter dropped usage trends measurements and throttle billable users refresh

Part of Project Bulkhead. Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636122

What does this MR do and why?

Analytics::UsageTrends::CounterJobWorker counts one usage trends measurement per job with Gitlab::Database::BatchCounter. Two of the counter's outcomes, :bad_config (the id range is too wide for the allowed number of batches) and :cancelled (Postgres kept cancelling the batch query after the batch size was halved down to the minimum), return normally with no row written, no job queued and no retry. A measurement dropped this way is indistinguishable from one that was never scheduled.

The admin endpoint PUT /license/:id/refresh_billable_users queues a full billable users count over the whole users id range on every call. The only thing collapsing repeated calls is job deduplication keyed on recorded_at, which spans one second, so one admin can start one count chain per second.

This MR adds the two controls the issue proposes:

1. The worker logs each dropped measurement on the job's done line (identifier, outcome, id range, recorded_at) and increments gitlab_usage_trends_measurement_dropped_total{reason}. Counting, the row insert and the continuation on timeout are unchanged. The batch counter's caps are not touched.
2. The endpoint is rate limited through Gitlab::ApplicationRateLimiter with a new refresh_billable_users key, returning the standard 429 with Retry-After. The check runs after the license lookup so a 404 does not consume the slot.

Business impact

Usage Trends charts (admin area) are fed by a nightly job that counts one table per measurement in batches. Two outcomes of that counter, bad_config and cancelled, end the job with no row, no retry and no error, so a data point that was dropped is indistinguishable from one that was never scheduled and the chart quietly goes stale. Separately, PUT /license/:id/refresh_billable_users starts a full count of billable users over the whole users table on every call, and job deduplication only collapses calls made in the same second, so one admin with a script can start a new full-table count chain every second.

With this MR every dropped measurement is logged on the Sidekiq done line (json.extra.analytics_usage_trends_counter_job_worker.drop_reason) and counted (gitlab_usage_trends_measurement_dropped_total{reason}), so a measurement that stops landing can be alerted on, and the refresh endpoint accepts one call per admin every 10 minutes. The second call within the window returns HTTP 429 with Retry-After: 600 and queues no job; other admins are unaffected; a non-existent licence still returns 404 without spending the slot. This protects the shared PostgreSQL database from repeated full-table counts and gives Optimize and Infrastructure a signal when Usage Trends data is missing. The throttle is behind throttle_refresh_billable_users (disabled by default); the metering ships unflagged because it only adds a log line and a counter on paths that today write nothing.

Queries this change generates

No new or changed SQL. The batch count and the analytics_usage_trends_measurements insert are unchanged; the rate limit is a Redis check through Gitlab::ApplicationRateLimiter (rule limit_billable_users_refreshes_by_user, derived from code), and it runs after the existing licenses primary-key lookup so a 404 does not consume the slot.

Survivable threshold

One refresh per admin user every 10 minutes. A refresh is at most one COUNT per 100,000 ids over the users table, and the nightly cron already runs the same count once a day; ten minutes is long enough for the previous chain to finish on an instance of any size, and short enough that an admin checking a seat count after a license change is not blocked for a meaningful time. The limit is keyed on the user only: the endpoint is admin-only, and the cost is the same whichever license id is passed.

Rollout

• The throttle is behind throttle_refresh_billable_users (gitlab_com_derisk, disabled by default). Rollout issue: #637726
• Rollback is turning the flag off; the endpoint then behaves exactly as before.
• The metering in the worker ships unflagged. It only adds done-line metadata and a counter increment on paths that today write nothing.

Verification

• spec/workers/analytics/usage_trends/counter_job_worker_spec.rb: the done-line metadata and the counter for :bad_config and :cancelled, and neither for a completed count.
• ee/spec/requests/api/license_spec.rb: the second call by the same admin within the window returns 429 with Retry-After and queues no job, a second admin is not affected, and with the flag disabled every call is accepted.
• ee/spec/lib/ee/gitlab/application_rate_limiter/labkit_adapter/supported_rate_limits_spec.rb and ee/spec/lib/ee/gitlab/application_rate_limiter_spec.rb: the new key and its threshold and period.

Local run of the four files: 160 examples, 0 failures, 14 pending (pre-existing shared examples).

Follow-ups

• Enable the flag on staging and production through the rollout issue, then remove it and the "Disabled by default" note from the rate limits documentation.
• Whether some measurements are already hitting :bad_config on GitLab.com is not settled by this MR; a production check is tracked in the issue.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.