Cap audit streaming destinations and rate limit their mutations

What changes for users

• With enforce_audit_streaming_destination_limits on, a group can have at most 50 audit streaming destinations in total, active or not. Existing destinations keep working.
• With the flag on, group destination create and update mutations allow 10 calls per user per minute. This covers groupAuditEventStreamingDestinationsCreate and groupAuditEventStreamingDestinationsUpdate, and the 6 legacy group mutations (externalAuditEventDestinationCreate/Update, googleCloudLoggingConfigurationCreate/Update, auditEventsAmazonS3ConfigurationCreate/Update). Instance-level mutations are not affected.

Why

The plan limit on destinations only counts active rows, so inactive rows have no bound. The create and update mutations also have no rate limit. The legacy group mutations are included so the limit can't be bypassed while they exist. Removal of the legacy APIs moved to a follow-up MR: !268077 "Draft: Remove legacy group audit streaming GraphQL APIs" !268077

The change

• Total cap. A validation in ee/app/models/concerns/audit_events/activatable.rb (TOTAL_LIMIT = 50, create only) counts all rows per group, inactive included. The legacy models get it through the same shared concern.
• Rate limit. 10 calls per minute per user on the 2 new and 6 legacy group create and update mutations. Over-limit calls are logged to auth_json.log, including calls without a request (GraphQL subscriptions, MCP tools). Those log the same Application_Rate_Limiter_Request message with the standard LabKit user fields (gl_user_id, gl_user_name). Calls with a request go through the core limiter, which logs user_id and username.
• Feature flag. enforce_audit_streaming_destination_limits (gitlab_com_derisk, default off, group actor) gates the cap and the rate limit. While off, they only log. Rollout issue: #638126 "[FF] enforce_audit_streaming_destination_limits -- Enforce the audit streaming destination cap and mutation rate limit". Plan: log-only for about a week, enable on GitLab.com, then remove the flag. Rollback is disabling the flag.
• Docs. doc/administration/instance_limits.md documents the 5 active destinations in total (all three types combined), the 50 total cap and the rate limit, with a feature flag note. doc/user/compliance/audit_event_streaming.md links to it.

Database

The cap adds one bounded count query per destination create, on four tables, with no schema changes.

SELECT COUNT(*) FROM (SELECT 1 AS one FROM "audit_events_group_external_streaming_destinations" WHERE "audit_events_group_external_streaming_destinations"."group_id" = <group_id> LIMIT 50) subquery_for_count;
SELECT COUNT(*) FROM (SELECT 1 AS one FROM "audit_events_external_audit_event_destinations" WHERE "audit_events_external_audit_event_destinations"."namespace_id" = <group_id> LIMIT 50) subquery_for_count;
SELECT COUNT(*) FROM (SELECT 1 AS one FROM "audit_events_amazon_s3_configurations" WHERE "audit_events_amazon_s3_configurations"."namespace_id" = <group_id> LIMIT 50) subquery_for_count;
SELECT COUNT(*) FROM (SELECT 1 AS one FROM "audit_events_google_cloud_logging_configurations" WHERE "audit_events_google_cloud_logging_configurations"."namespace_id" = <group_id> LIMIT 50) subquery_for_count;

Table	Index (Index Only Scan, 0 heap fetches)	Plan (fixed group)	Plan (busiest group)
audit_events_group_external_streaming_destinations	unique_idx_group_destinations_on_name_category_group	0.6 ms	9.2 ms
audit_events_external_audit_event_destinations	index_audit_events_external_audit_on_ns_verification_token	1.9 ms	6.4 ms
audit_events_amazon_s3_configurations	unique_amazon_s3_configurations_namespace_id_and_bucket_name	0.6 ms	5.0 ms
audit_events_google_cloud_logging_configurations	uniq_google_cloud_logging_configuration_namespace_id_and_name	1.4 ms	0.7 ms

The busiest-group plans include a subquery that picks the group with the most rows. That subquery is only for the measurement and accounts for most of the time.

Customer impact

The limits affect no one. The largest group has 11 destinations, and the new create mutation sees 1 to 4 calls a day and update 0 to 9.

How to validate

In a Rails console, as an owner of a top-level Ultimate group. The script runs both flag states itself and rolls back all writes:

root = User.find_by_username('root')
group = Group.top_level.find { |g| g.owned_by?(root) && g.licensed_feature_available?(:external_audit_events) }
puts "group=#{group.full_path}"
new_update = <<~GQL
  mutation($id: AuditEventsGroupExternalStreamingDestinationID!, $name: String) {
    groupAuditEventStreamingDestinationsUpdate(input: { id: $id, name: $name }) { errors }
  }
GQL
legacy_update = <<~GQL
  mutation($id: AuditEventsExternalAuditEventDestinationID!, $url: String) {
    externalAuditEventDestinationUpdate(input: { id: $id, destinationUrl: $url }) { errors }
  }
GQL
[false, true].each do |flag|
  flag ? Feature.enable(:enforce_audit_streaming_destination_limits, group) : Feature.disable(:enforce_audit_streaming_destination_limits)
  Gitlab::Redis::RateLimiting.with { |r| r.scan_each(match: '*audit_event_streaming_destination_mutation*') { |k| r.del(k) } }
  ApplicationRecord.transaction do
    existing = AuditEvents::Group::ExternalStreamingDestination.where(group: group).count
    (50 - existing).times do |i|
      AuditEvents::Group::ExternalStreamingDestination.create!(group: group, name: "cap-check-#{i}", category: 'http',
        config: { 'url' => "https://example.com/cap/#{i}" }, active: false)
    end
    extra = AuditEvents::Group::ExternalStreamingDestination.new(group: group, name: 'cap-check-50', category: 'http',
      config: { 'url' => 'https://example.com/cap/50' })
    puts "flag=#{flag} 51st valid=#{extra.valid?} #{extra.errors.full_messages.inspect}"
    dest = AuditEvents::Group::ExternalStreamingDestination.where(group: group).first
    r = 11.times.map do
      GitlabSchema.execute(new_update, variables: { 'id' => dest.to_global_id.to_s, 'name' => dest.name },
        context: { current_user: root }).to_h.then { |h| h.dig('data', 'groupAuditEventStreamingDestinationsUpdate', 'errors') || h['errors'] }
    end
    puts "flag=#{flag} new updates: #{r.map { |e| e.blank? ? 'ok' : e.inspect }.tally}"
    Gitlab::Redis::RateLimiting.with { |rr| rr.scan_each(match: '*audit_event_streaming_destination_mutation*') { |k| rr.del(k) } }
    legacy = AuditEvents::ExternalAuditEventDestination.create!(group: group, destination_url: 'https://example.com/legacy-check')
    r = 11.times.map do
      GitlabSchema.execute(legacy_update, variables: { 'id' => legacy.to_global_id.to_s, 'url' => legacy.destination_url },
        context: { current_user: root }).to_h.then { |h| h.dig('data', 'externalAuditEventDestinationUpdate', 'errors') || h['errors'] }
    end
    puts "flag=#{flag} legacy updates: #{r.map { |e| e.blank? ? 'ok' : e.inspect }.tally}"
    raise ActiveRecord::Rollback
  end
end
Feature.remove(:enforce_audit_streaming_destination_limits)
puts "flag removed; auth_json.log throttle lines: #{`grep -c audit_event_streaming_destination_mutation_request_limit log/auth_json.log 2>/dev/null`.strip}"

Results (on master everything succeeds, since the cap and rate limit don't exist there):

Check	This branch, flag off	This branch, flag on
51st destination (50 inactive exist)	valid (logged)	invalid: "Maximum number of external audit event destinations (50) exceeded, including inactive ones. Delete unused destinations to add more."
11 new-mutation updates by one user in a minute	all 11 succeed (logged to auth_json.log)	10 succeed, the 11th returns "This endpoint has been requested too many times. Try again later."
11 legacy-mutation updates (externalAuditEventDestinationUpdate) in a minute	all 11 succeed (logged)	10 succeed, the 11th returns the same message

Related to #636584 "Audit streaming destination mutations (caps, limits, length validation)"