Paginate the policy store list endpoint

Marcos Rocharequested to merge
608267-paginate-policy-store-list-endpoint into master

What does this MR do and why?

This MR paginates GET /organizations/:id/security/policy_store. Gitlab::PolicyStore::Ports::PolicyRepository#list now returns a Gitlab::PolicyStore::Page instead of a plain Array of every policy the organization owns. This was deferred in !248603 (merged), pending a port-level decision.

Callers get a bounded page instead of the full list: 20 policies by default (DEFAULT_PER_PAGE), capped at 100 (MAX_PER_PAGE). The response carries X-Page, X-Per-Page, X-Next-Page, and X-Prev-Page headers, with no X-Total or X-Total-Pages (see Design decisions). A negative page or per_page returns 400 Bad Request, matching sibling paginated endpoints. The list keeps the full Policy entity, since auth/glaz's in-flight REST client (auth/glaz!139) reads scope_rego and the compiled policy_rego from this same route.

GraphQL pagination is out of scope for this MR.

Design decisions

The response carries no exact total count, by design.

Offset-based pagination is the easiest way to paginate over records. However, it does not scale well for large database tables. As a long-term solution, keyset pagination is preferred.

Avoid presenting total counts, prefer limit counts.

Source: Pagination guidelines: Prepare for scaling

A full rewrite to keyset pagination is a bigger contract change than this issue asked for. Instead, both adapters fetch one row past per_page and use its presence to answer has_next_page?, the same technique Kaminari's without_count mode and Govern::Policy.evaluation_candidates already use in this codebase. This also avoids the count-vs-fetch race a COUNT(*) design would carry, where two independent queries could disagree under concurrent writes.

The port speaks offset, not page number. A page number is a caller-facing framing. ListService translates it to an offset, clamping the incoming page to its own MAX_PAGE (1,000). The port separately clamps offset to MAX_OFFSET (100,000) and per_page to MAX_PER_PAGE (100), since Gitlab::PolicyStore is the only entry point callers are supposed to use. This is defense in depth, not redundancy.

ids: bypasses pagination rather than filtering a fetched page. #list now filters by primary key at the query level when ids is given, since the caller already knows the exact, bounded set of policies it wants. Filtering the already-paginated result instead would silently drop a requested id that falls outside the current page.

Gitlab::PolicyStore::Page, the return value of #list, behaves as an Enumerable over the policies it holds. This lets callers that only care about the policy list, not the pagination metadata, keep treating a Page like the plain Array #list used to return.

Query plans

govern_policies has no production rows. Seeded 20,000 rows across 50 organizations, 500 namespaces, and three trigger types (one row in ten given no namespace), following the recipe from !249574 (merged).

Govern::Policy.paginated's single query, fetching per_page + 1 rows:

-- no trigger_type, per_page 20 (fetches 21)
SELECT "govern_policies".* FROM "govern_policies"
WHERE "govern_policies"."organization_id" = 1
ORDER BY "govern_policies"."id" ASC LIMIT 21 OFFSET 0;

plan

-- with trigger_type, per_page 20 (fetches 21)
SELECT "govern_policies".* FROM "govern_policies"
WHERE "govern_policies"."organization_id" = 1 AND "govern_policies"."trigger_type" = 0
ORDER BY "govern_policies"."id" ASC LIMIT 21 OFFSET 0;

plan

When ids is given, #list bypasses paginated entirely and filters by primary key instead, through ApplicationRecord.id_in:

-- ids only
SELECT "govern_policies".* FROM "govern_policies"
WHERE "govern_policies"."organization_id" = 1 AND "govern_policies"."id" IN (50, 100, 150)
ORDER BY "govern_policies"."id" ASC;

plan

-- ids combined with trigger_type
SELECT "govern_policies".* FROM "govern_policies"
WHERE "govern_policies"."organization_id" = 1 AND "govern_policies"."trigger_type" = 0
  AND "govern_policies"."id" IN (150, 300, 450)
ORDER BY "govern_policies"."id" ASC;

plan

Flagging for the database reviewer rather than asserting the existing indexes are sufficient at scale.

How to set up and validate locally

  1. Enable the experiment on the rails console. Requires an Ultimate license.
Feature.enable(:security_policies_v2)
Gitlab::CurrentSettings.current_application_settings.update!(policy_store_experiment_enabled: true)

Verify both took effect, since the endpoint's own before block reads exactly these two predicates:

Feature.enabled?(:security_policies_v2, :instance) # => true
Gitlab::CurrentSettings.policy_store_experiment_enabled? # => true
  1. As an organization owner, create three policies through the REST API:
curl --include --request POST --header "PRIVATE-TOKEN: <your_access_token>" \
  --data 'name=Policy A&trigger_type=deployment_requested&rules[][type]=custom&rules[][value]=package governance' \
  --url "http://gdk.test:3000/api/v4/organizations/1/security/policy_store"

Verify each of the three requests returns 201 Created, so a feature flag or license miss in step 1 surfaces here rather than as a confusing empty list in step 3. Repeat with name=Policy B and name=Policy C.

  1. List with the default page size:
curl --include --header "PRIVATE-TOKEN: <your_access_token>" \
  --url "http://gdk.test:3000/api/v4/organizations/1/security/policy_store"

Verify the response headers read X-Page: 1, X-Per-Page: 20, X-Next-Page empty (no more policies to fetch), and that there is no X-Total or X-Total-Pages header.

  1. List one page at a time:
curl --include --header "PRIVATE-TOKEN: <your_access_token>" \
  --url "http://gdk.test:3000/api/v4/organizations/1/security/policy_store?per_page=1"

Verify the body has exactly one policy and X-Next-Page reads 2. Add &page=2 to the same request and verify it returns the next policy in id order (not the same one), with X-Next-Page now empty and X-Prev-Page: 1.

  1. Request an oversized page:
curl --include --header "PRIVATE-TOKEN: <your_access_token>" \
  --url "http://gdk.test:3000/api/v4/organizations/1/security/policy_store?per_page=1000"

Verify X-Per-Page reads 100, not 1000, because per_page is clamped to MAX_PER_PAGE rather than honored as requested.

References

Edited by Marcos Rocha

Merge request reports