Memoize policy class maps during Orbit redaction batches

What does this MR do and why?

POST /api/:version/internal/orbit/redaction allocates about 3 MB and spends about 8 ms of CPU per project ID. A 1001-ID batch is 3 GB and 6 to 12 s, almost all Ruby CPU. SQL is under 2%.

The cost is in the declarative_policy gem. DeclarativePolicy::Base.conditions rebuilds superclass.conditions.merge(own_conditions) on every call, and each new condition object calls it twice. ProjectPolicy has 799 conditions, so one Ability.allowed? copies that hash about 40 times.

Two changes, both inside the redaction layer:

  1. Authz::RedactionPolicyClassMaps memoizes the four policy class maps per class while Authz::RedactionService runs its checks. The memo lives in SafeRequestStore for the batch and is removed when the batch ends; nested batches share it. Outside a batch the patch calls the gem method, so no other caller changes.
  2. EE::Authz::RedactionService preloads root ancestors, saml_provider and ip_restrictions for the group namespaces in the batch. ProjectRootAncestorPreloader returns early for an Array, which this service passes, so the EE policy preloader ran one namespaces and one saml_providers query per root group.

Authorization results do not change. A sweep of all 752 project abilities on a 300-root-group dataset shows zero result changes and no ability with more SQL.

Numbers

GDK VM, read_code, 1000 projects, thread CPU, 3 runs.

Scenario Build Allocated per ID CPU per ID SQL per batch
2 root groups master 2.5 MB 3.4 ms 23
2 root groups this MR 69 KB 0.62 ms 21
300 root groups master 2.6 MB 3.6 ms 624
300 root groups this MR 105 KB 0.92 ms 21

Production runs about 2.4x the VM CPU per ID. A 1001-ID request should drop from about 8 s and 3 GB to about 2 s and 100 MB. Prod correlation IDs, the full SQL list, and the reply to the questions on !239362 (closed) are in the first comment.

How to set up and validate locally

bundle exec rspec spec/lib/authz/redaction_policy_class_maps_spec.rb \
  spec/services/authz/redaction_service_spec.rb \
  ee/spec/services/ee/authz/redaction_service_spec.rb

The EE N+1 example fails on master and passes here.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Michael Angelo Rivera

Merge request reports

Loading
Loading