Integrate the policy store gem with the Govern models (!248000) · Merge requests · GitLab.org / GitLab

What does this MR do and why?

Backs the Policy Store with the govern_policies table added in !247753 (merged) (merged), replacing the gem's in-memory default with Govern::PolicyStore::ActiveRecordPolicyRepository on EE. Nothing user-facing changes: the only callers are the Security::SecurityOrchestrationPolicies::PolicyStore services already on master, and every one of them sits behind policy_store_experiment_active?.

The adapter implements the existing port unchanged, so create, find, delete and list(organization_id:) keep the names and the ValidationError contract that those four services already call. It reuses the port's own normalize_attributes, validate_required_attributes! and with_compiled_scope, which means scope transpilation behaves identically whether the store is in memory or in Postgres. Records are converted to frozen Gitlab::PolicyStore::Policy value objects at the boundary, with jsonb deep-duped, so no ActiveRecord object leaves the component. config/initializers/policy_store.rb injects it on EE only, in a reload-safe to_prepare; FOSS keeps the gem's inert in-memory default.

The port gains two operations. update(id, attributes) bumps version by exactly one under with_lock and ignores identity and tenancy attributes. list_for_evaluation(organization_id:, trigger_id:) is the engine fetch path per GOVERN-006: an organization's active policies for one trigger, capped at Govern::Policy::EVALUATION_LIMIT. It is deliberately separate from list, because evaluation must never see disabled policies while management surfaces must see all of them. Both are covered by the gem's 'a policy repository' shared examples, which the ActiveRecord adapter now runs against the real table alongside the in-memory adapter.

The adapter is not an authorization boundary: find and list return any policy by id or organization. Permission checks belong to the calling service layer (#606281 (closed)).

Design decisions

namespace_id becomes nullable, which was not the plan when the base MR merged. A policy will not always belong to a group: organization-owned policies have no owning namespace. That is also the answer to the sharding-key question raised in review on the base MR (!247753 (comment 3654948707)), so organization_id stays the sharding key and the table keeps its organization transfer support entry.

Postgres treats NULLs as distinct, so the unique index on (organization_id, namespace_id, name) would stop constraining organization-owned rows against each other entirely. It is replaced by two partial unique indexes, one per ownership form. Neither can serve the sharding key on its own, so the migration also adds a composite index leading with organization_id, which covers both the sharding key and the engine fetch.

organization_id is supplied through the port rather than derived from the namespace, because an organization-owned policy has no namespace to derive it from. The namespace_matches_organization model validation still rejects a mismatched pair.

Database

govern_policies lives in the gitlab_sec schema (db/docs/govern_policies.yml), so the owning database is sec. DDL runs against all three databases, but sec is the one that matters here and it is the one whose output is quoted below. The table carries no foreign keys: organization_id and namespace_id reach gitlab_main_org through the loose foreign keys added in the base MR, so nothing here can join or cascade across schemas.

One migration on a new, empty, unwritten table.

Migration	Contents
`make_namespace_id_nullable_on_govern_policies`	`change_column_null` on `namespace_id`; swaps `unique_govern_policies_organization_id_namespace_id_and_name` for two partial unique indexes; adds a composite index covering the engine fetch

Indexes after the migration:

Index	Columns	Unique	Predicate
`unique_govern_policies_org_namespace_and_name`	`(organization_id, namespace_id, name)`	yes	`namespace_id IS NOT NULL`
`unique_govern_policies_org_and_name_without_namespace`	`(organization_id, name)`	yes	`namespace_id IS NULL`
`index_govern_policies_on_org_trigger_lifecycle_and_id`	`(organization_id, trigger_type, lifecycle_state, id)`	no
`index_govern_policies_on_namespace_id`	`(namespace_id)`	no

The two partial predicates are exhaustive and disjoint, so every row is covered by exactly one uniqueness rule. The composite index covers the engine fetch below, and because it leads with organization_id it also serves the sharding key, which neither partial index can do on its own. A separate single-column index on organization_id would be redundant against it.

Queries

The table is empty in production, so these have no meaningful plan yet. The seed block below populates it so the plans can be evaluated in the database tool. Run everything against the sec database.

New reads, exactly as ActiveRecord emits them:

-- ActiveRecordPolicyRepository#list(organization_id:)
SELECT "govern_policies".* FROM "govern_policies"
 WHERE "govern_policies"."organization_id" = 1
 ORDER BY "govern_policies"."id" ASC;

-- ActiveRecordPolicyRepository#list_for_evaluation(organization_id:, trigger_id:)
-- Govern::Policy.evaluation_candidates, the GOVERN-006 engine fetch path
SELECT "govern_policies".* FROM "govern_policies"
 WHERE "govern_policies"."organization_id" = 1
   AND "govern_policies"."lifecycle_state" = 0
   AND "govern_policies"."trigger_type" = 0
 ORDER BY "govern_policies"."id" ASC
 LIMIT 101;

The limit is EVALUATION_LIMIT + 1, not EVALUATION_LIMIT. The extra row is how the adapter tells a full page from a truncated one, so that dropping an organization's newest policies out of enforcement reports to error tracking instead of passing silently.

-- ActiveRecordPolicyRepository#find(id) and the read inside #update / #delete
SELECT "govern_policies".* FROM "govern_policies" WHERE "govern_policies"."id" = 1 LIMIT 1;

-- The row lock taken by #update, via record.with_lock
SELECT "govern_policies".* FROM "govern_policies" WHERE "govern_policies"."id" = 1 LIMIT 1 FOR UPDATE;

-- Uniqueness validation on save, organization-owned policy (namespace_id IS NULL)
SELECT "govern_policies".* FROM "govern_policies"
 WHERE "govern_policies"."name" = 'x'
   AND "govern_policies"."organization_id" = 1
   AND "govern_policies"."namespace_id" IS NULL
 LIMIT 1;

-- Uniqueness validation on save, group-owned policy
SELECT "govern_policies".* FROM "govern_policies"
 WHERE "govern_policies"."name" = 'x'
   AND "govern_policies"."organization_id" = 1
   AND "govern_policies"."namespace_id" = 7
 LIMIT 1;

Writes are all single-row: an INSERT on create, an UPDATE ... WHERE id = ? on update, and a DELETE ... WHERE id = ? on delete. No bulk or batched operation is added.

Seed data for plan evaluation

govern_policies has no foreign keys, so these inserts need no matching organizations or namespaces rows. 500 organizations of 1000 policies each, split across the three trigger types and both lifecycle states, which puts roughly 165 rows behind each (organization_id, trigger_type, lifecycle_state) combination and so exercises the LIMIT on the evaluation query. Adjust the two moduli to reshape the distribution.

INSERT INTO govern_policies (
  organization_id, namespace_id, created_at, updated_at,
  version, trigger_type, mode, lifecycle_state, name, rules, actions
)
SELECT
  (series % 500) + 1                                                    AS organization_id,
  CASE WHEN series % 4 = 0 THEN NULL ELSE (series % 5000) + 1 END       AS namespace_id,
  now(), now(),
  1,
  (series % 3)::smallint                                                AS trigger_type,
  2::smallint                                                           AS mode,
  (series % 2)::smallint                                                AS lifecycle_state,
  'seed-policy-' || series                                              AS name,
  '[]'::jsonb, '[]'::jsonb
FROM generate_series(1, 500000) AS series;

ANALYZE govern_policies;

Names are globally unique in the seed, so neither partial unique index rejects a row. To clean up: DELETE FROM govern_policies WHERE name LIKE 'seed-policy-%';

The plans below were taken locally at 20,000 rows, so lower the generate_series bound if you want to reproduce them exactly. The timings at that size mean nothing; the plan shape is the point. Both were measured against this MR, comparing a single-column organization_id index against the composite one, since govern_policies had no index on organization_id at all before this MR and so has no plan on master to compare with.

With only organization_id indexed, the other two predicates were applied as a filter and the ordering needed a sort:

Limit  (cost=40.26..40.28 rows=7) (actual rows=13 loops=1)
  Buffers: shared hit=45
  ->  Sort
        Sort Key: id
        ->  Index Scan using index_govern_policies_on_organization_id on govern_policies
              Index Cond: (organization_id = 1)
              Filter: ((lifecycle_state = 0) AND (trigger_type = 0))
              Rows Removed by Filter: 27

With (organization_id, trigger_type, lifecycle_state, id) all three predicates become index conditions, the sort disappears because the index is already in id order within the matched prefix, and buffer hits drop from 45 to 21:

Limit  (cost=0.29..8.44 rows=7) (actual rows=13 loops=1)
  Buffers: shared hit=21
  ->  Index Scan using index_govern_policies_on_org_trigger_lifecycle_and_id on govern_policies
        Index Cond: ((organization_id = 1) AND (trigger_type = 0) AND (lifecycle_state = 0))

list still sorts, because the composite index orders by id only within a (trigger_type, lifecycle_state) prefix and list spans all of them. That is not a regression: a single-column organization_id index produces the same plan and the same 45 buffer hits.

Sort  (cost=41.04..41.14 rows=40) (actual rows=40 loops=1)
  Sort Key: id
  Buffers: shared hit=45
  ->  Index Scan using index_govern_policies_on_org_trigger_lifecycle_and_id on govern_policies
        Index Cond: (organization_id = 1)

Please confirm both shapes hold at production volume. list is unbounded by design, so its cost scales with policies-per-organization; pagination lands with the first API caller.

Migration output

Rollback verified locally on all three databases: db:migrate:down reverts all four index operations and restores the NOT NULL constraint, and db:migrate:up reapplies cleanly.

Note that down is only reversible while no organization-owned policies exist. Once a row has namespace_id IS NULL, change_column_null(..., false) raises a NOT NULL violation. That is inherent to relaxing the column, and the table ships dark, so no rollback path is affected today.

db:migrate:up:sec output

sec: == 20260806120000 MakeNamespaceIdNullableOnGovernPolicies: migrating ==========
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0404s
sec: -- index_exists?(:govern_policies, [:organization_id, :namespace_id, :name], {:unique=>true, :where=>"namespace_id IS NOT NULL", :name=>"unique_govern_policies_org_namespace_and_name", :algorithm=>:concurrently})
sec:    -> 0.0023s
sec: -- execute("SET statement_timeout TO 0")
sec:    -> 0.0003s
sec: -- add_index(:govern_policies, [:organization_id, :namespace_id, :name], {:unique=>true, :where=>"namespace_id IS NOT NULL", :name=>"unique_govern_policies_org_namespace_and_name", :algorithm=>:concurrently})
sec:    -> 0.0040s
sec: -- execute("RESET statement_timeout")
sec:    -> 0.0004s
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0004s
sec: -- index_exists?(:govern_policies, [:organization_id, :name], {:unique=>true, :where=>"namespace_id IS NULL", :name=>"unique_govern_policies_org_and_name_without_namespace", :algorithm=>:concurrently})
sec:    -> 0.0016s
sec: -- add_index(:govern_policies, [:organization_id, :name], {:unique=>true, :where=>"namespace_id IS NULL", :name=>"unique_govern_policies_org_and_name_without_namespace", :algorithm=>:concurrently})
sec:    -> 0.0026s
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0003s
sec: -- index_exists?(:govern_policies, [:organization_id, :trigger_type, :lifecycle_state, :id], {:name=>"index_govern_policies_on_org_trigger_lifecycle_and_id", :algorithm=>:concurrently})
sec:    -> 0.0017s
sec: -- add_index(:govern_policies, [:organization_id, :trigger_type, :lifecycle_state, :id], {:name=>"index_govern_policies_on_org_trigger_lifecycle_and_id", :algorithm=>:concurrently})
sec:    -> 0.0016s
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0002s
sec: -- index_name_exists?(:govern_policies, "unique_govern_policies_organization_id_namespace_id_and_name")
sec:    -> 0.0005s
sec: -- remove_index(:govern_policies, {:algorithm=>:concurrently, :name=>"unique_govern_policies_organization_id_namespace_id_and_name"})
sec:    -> 0.0022s
sec: -- change_column_null(:govern_policies, :namespace_id, true, nil)
sec:    -> 0.0004s
sec: == 20260806120000 MakeNamespaceIdNullableOnGovernPolicies: migrated (0.1020s) =

db:migrate:down:sec output

sec: == 20260806120000 MakeNamespaceIdNullableOnGovernPolicies: reverting ==========
sec: -- change_column_null(:govern_policies, :namespace_id, false, nil)
sec:    -> 0.0014s
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0343s
sec: -- index_exists?(:govern_policies, [:organization_id, :namespace_id, :name], {:unique=>true, :name=>"unique_govern_policies_organization_id_namespace_id_and_name", :algorithm=>:concurrently})
sec:    -> 0.0024s
sec: -- execute("SET statement_timeout TO 0")
sec:    -> 0.0003s
sec: -- add_index(:govern_policies, [:organization_id, :namespace_id, :name], {:unique=>true, :name=>"unique_govern_policies_organization_id_namespace_id_and_name", :algorithm=>:concurrently})
sec:    -> 0.0024s
sec: -- execute("RESET statement_timeout")
sec:    -> 0.0003s
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0002s
sec: -- index_name_exists?(:govern_policies, "index_govern_policies_on_org_trigger_lifecycle_and_id")
sec:    -> 0.0005s
sec: -- remove_index(:govern_policies, {:algorithm=>:concurrently, :name=>"index_govern_policies_on_org_trigger_lifecycle_and_id"})
sec:    -> 0.0015s
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0002s
sec: -- index_name_exists?(:govern_policies, "unique_govern_policies_org_and_name_without_namespace")
sec:    -> 0.0004s
sec: -- remove_index(:govern_policies, {:algorithm=>:concurrently, :name=>"unique_govern_policies_org_and_name_without_namespace"})
sec:    -> 0.0006s
sec: -- transaction_open?(nil)
sec:    -> 0.0000s
sec: -- view_exists?(:postgres_partitions)
sec:    -> 0.0003s
sec: -- index_name_exists?(:govern_policies, "unique_govern_policies_org_namespace_and_name")
sec:    -> 0.0005s
sec: -- remove_index(:govern_policies, {:algorithm=>:concurrently, :name=>"unique_govern_policies_org_namespace_and_name"})
sec:    -> 0.0007s
sec: == 20260806120000 MakeNamespaceIdNullableOnGovernPolicies: reverted (0.0824s) =

How to set up and validate locally

Requires an Ultimate licence, because policy_store_experiment_available? checks licensed_feature_available?(:security_orchestration_policies). The gate reads five things in total, so step 2 asserts the whole predicate rather than trusting the individual settings.

Enable the experiment for a top-level group on the rails console

group = Group.first
Feature.enable(:security_policies_v2)
ApplicationSetting.current.update!(policy_store_experiment_enabled: true)
group.namespace_settings.update!(policy_store_experiment_enabled: true)

Confirm the gate is actually satisfied, because enabling a setting is not the same as passing the check

group.root?                                # => true
group.policy_store_experiment_available?   # => true
group.policy_store_experiment_active?      # => true

Confirm the facade is wired to Postgres rather than the in-memory default

Gitlab::PolicyStore.configuration.repository
# => #<Govern::PolicyStore::ActiveRecordPolicyRepository ...>

Create a group-owned policy and confirm it persisted, with scope_rego compiled from the authored policy_scope

def show(result)
  puts(result.success? ? result.payload[:policy].to_h : result.message)
end

result = Security::SecurityOrchestrationPolicies::PolicyStore::CreateService.new(
  group: group,
  params: {
    name: 'Deployment gate',
    trigger_id: 'deployment_requested',
    rules: [{ 'type' => 'custom', 'value' => 'package governance' }],
    actions: [{ 'type' => 'require_approval' }],
    policy_scope: { 'compliance_frameworks' => [{ 'id' => 1 }] }
  }
).execute

show(result)

Verify the row is really in the database and not in a process-local hash, which is what this MR changes

persisted = Govern::Policy.find_by(name: 'Deployment gate')

persisted.present?            # => true
persisted.namespace_id        # => nil, because CreateService is organization-scoped
persisted.organization_id     # => group.organization_id
persisted.scope_rego          # => the compiled "package gitlab.scope" program

Create an organization-owned policy directly through the port, then a group-owned one with the same name, and verify both are accepted, because the two partial unique indexes constrain the two ownership forms separately

repository = Gitlab::PolicyStore.configuration.repository

organization_owned = repository.create(
  organization_id: group.organization_id, name: 'Shared name', trigger_id: 'deployment_requested')
group_owned = repository.create(
  organization_id: group.organization_id, namespace_id: group.id,
  name: 'Shared name', trigger_id: 'deployment_requested')

[organization_owned.namespace_id, group_owned.namespace_id] # => [nil, group.id]

Verify a second organization-owned policy of that name is rejected, because unique_govern_policies_org_and_name_without_namespace covers the null-namespace rows

repository.create(
  organization_id: group.organization_id, name: 'Shared name', trigger_id: 'deployment_requested')
# => Gitlab::PolicyStore::ValidationError: Name has already been taken

Update a policy and verify the version moves by exactly one while the organization does not, because identity and tenancy attributes are ignored

updated = repository.update(group_owned.id, name: 'Renamed', organization_id: -1)

[group_owned.version, updated.version]                       # => [1, 2]
[group_owned.organization_id, updated.organization_id]       # => equal

Verify the evaluation read returns only the organization's active policies for the trigger

repository.update(organization_owned.id, lifecycle_state: 'disabled')

Gitlab::PolicyStore.list_for_evaluation(
  organization_id: group.organization_id, trigger_id: 'deployment_requested').map(&:name)
# => the active policies only; 'Shared name' with a nil namespace is gone

Rename a policy whose scope was compiled from policy_scope, and verify the stored Rego follows the new name, because the transpiler embeds the name in the program the engine evaluates

scoped = repository.create(
  organization_id: group.organization_id, name: 'Original name',
  trigger_id: 'deployment_requested',
  policy_scope: { 'compliance_frameworks' => [{ 'id' => 1 }] })

renamed = repository.update(scoped.id, name: 'Renamed policy')

renamed.scope_rego.include?('Renamed policy')  # => true
renamed.scope_rego.include?('Original name')   # => false

Rename a policy whose Rego was authored by hand, and verify it is left exactly as written, because there is no structured source to recompile from

authored_text = "package gitlab.scope\n\n# hand written"
authored = repository.create(
  organization_id: group.organization_id, name: 'Authored policy',
  trigger_id: 'deployment_requested', scope_rego: authored_text)

repository.update(authored.id, name: 'Authored renamed').scope_rego == authored_text # => true

Clear scope_rego on the scoped policy and verify it falls back to the structured scope rather than widening to every project

reverted = repository.update(scoped.id, scope_rego: nil)

reverted.policy_scope                                        # => the compliance_frameworks hash, still there
reverted.scope_rego.include?('no policy_scope')              # => false

Saturate the evaluation limit and verify the truncation is reported rather than passing silently

# Lower the cap rather than authoring 101 policies
Govern::Policy.send(:remove_const, :EVALUATION_LIMIT)
Govern::Policy.const_set(:EVALUATION_LIMIT, 2)

3.times do |number|
  repository.create(organization_id: group.organization_id, name: "extra-#{number}",
    trigger_id: 'deployment_requested')
end

Gitlab::PolicyStore.list_for_evaluation(
  organization_id: group.organization_id, trigger_id: 'deployment_requested').size
# => 2, and log/exceptions_json.log gains a Gitlab::PolicyStore::Error naming the cap

Flip the gate back off and verify the services refuse without touching the store

group.namespace_settings.update!(policy_store_experiment_enabled: false)

result = Security::SecurityOrchestrationPolicies::PolicyStore::ListService.new(group: group).execute
[result.error?, result.reason] # => [true, :experiment_not_active]

References

Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/604367
Schema and models: !247753 (merged) (merged)
Sharding-key review thread that made namespace_id nullable: !247753 (comment 3654948707)
Permission checks, deliberately out of scope: #606281 (closed)
Stacked on top of this MR: !248039 (closed) (open)
Epic: https://gitlab.com/groups/gitlab-org/-/epics/22027
ADR GOVERN-008: https://gitlab.com/gitlab-org/architecture/govern/design-doc/-/blob/main/decisions/008-policy-store-modular-monolith.md

Edited Aug 07, 2026 by Marcos Rocha

Integrate the policy store gem with the Govern models