Add Policy Store evaluation and violation tables
What does this MR do and why?
Adds two new tables and their models for the Policy Store (Policies v2 experiment): govern_policy_evaluations and govern_policy_violations.
Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/607735. This MR covers the storage part (1 of 3) of that issue, so the issue stays open until the follow-ups land: an internal write API that the Policy Engine calls after each evaluation, and the read APIs for the Policy Store UI (evaluation lists and per-policy statistics).
Nothing writes to these tables yet. The only changes to existing code are two new has_many associations on Govern::Policy (:evaluations and :violations) and a skipped_transfer_models entry in Organizations::Transfer::UsersService (see below), plus spec and config registrations. It builds on the base tables added in !247753 (merged) (govern_policies and govern_policy_enforcements).
The idea is to store one evaluation row per completed policy evaluation, in every enforcement mode including audit. Audit mode logs evaluations with zero user impact, so we can build retrospective impact analysis on top of it later. Violations are a separate table and not a flag on the evaluation row, because a single evaluation can produce multiple rule violations, and warn/audit outcomes also need the violation details recorded, not just enforce/deny. This data powers the Policy Store UI: the evaluations list and the per-policy statistics widgets (evaluations this week, totals, violation counts, verdict breakdown).
Schema
Table: govern_policy_evaluations
| Column | Type | Notes |
|---|---|---|
organization_id |
bigint, NOT NULL | Sharding key, loose foreign key to organizations (async_delete) |
govern_policy_id |
bigint, NOT NULL | Foreign key to govern_policies, ON DELETE CASCADE |
project_id |
bigint, NULL | Scope reference, no foreign key (see Design decisions) |
environment_id |
bigint, NULL | Scope reference, no foreign key (see Design decisions) |
user_id |
bigint, NULL | Principal who triggered the evaluation, no foreign key (see Design decisions) |
created_at |
timestamptz, NOT NULL | |
updated_at |
timestamptz, NOT NULL | |
evaluated_at |
timestamptz, NOT NULL | Engine-reported completion time of the evaluation |
policy_version |
integer, NOT NULL | Pins the evaluation to the policy's config version at evaluation time |
trigger_type |
smallint, NOT NULL | Enum, snapshot of the policy's trigger type at evaluation time: deployment_requested: 0, environment_advanced: 1, deployment_promoted: 2 |
mode |
smallint, NOT NULL | Enum, snapshot of the policy's enforcement mode at evaluation time: audit: 0, warn: 1, enforce: 2 |
verdict |
smallint, NOT NULL | Enum: allow: 0, deny: 1, require_approval: 2 |
Indexes: (govern_policy_id, evaluated_at) serves the foreign key, per-policy listing and time-window statistics. (organization_id, evaluated_at) serves org-scope listing and the loose FK cleanup. (project_id) is for project-scoped filtering.
Table: govern_policy_violations
| Column | Type | Notes |
|---|---|---|
organization_id |
bigint, NOT NULL | Sharding key, loose foreign key to organizations (async_delete) |
govern_policy_evaluation_id |
bigint, NOT NULL | Foreign key to govern_policy_evaluations, ON DELETE CASCADE |
govern_policy_id |
bigint, NOT NULL | Foreign key to govern_policies, ON DELETE CASCADE. Denormalized from the evaluation so per-policy violation counts don't need a join, and a model validation keeps it consistent |
details |
jsonb, NULL | Violated rule reference and reasons. Schema validation is deferred to the write API MR (see Design decisions) |
Indexes: foreign key index on govern_policy_evaluation_id. (govern_policy_id, created_at) serves violation counts per policy, with the foreign key column leading. (organization_id) for the loose FK cleanup.
Design decisions
trigger_type and mode snapshot the policy's values at evaluation time. Policy rows are updated in place (version bump, jsonb rules overwritten), so the snapshot on the evaluation row is the only point-in-time record of what was actually evaluated.
project_id, environment_id and user_id don't have foreign keys, per GOVERN-008: the Policy Store owns its referential integrity so it can be extracted as a standalone service later. They are also cross-database references (sec to main). These rows are audit-style records that are expected to outlive the entities they reference, similar to audit_events and the existing govern_policy_enforcements.project_id. The columns are allowlisted in spec/db/schema_spec.rb.
govern_policy_id on violations is denormalized (it's derivable via the evaluation), so that per-policy violation counts don't require a join through govern_policy_evaluations. A model validation enforces that it stays consistent with the parent evaluation's policy.
ON DELETE CASCADE from govern_policies means deleting a policy deletes its evaluation and violation history. This is consistent with the existing govern_policy_enforcements table and I think it's acceptable for the experiment. But if retrospective impact analysis (https://gitlab.com/gitlab-org/gitlab/-/work_items/607736) turns out to need the history to survive policy deletion, we can revisit this before GA.
details doesn't have a JsonSchemaValidator yet. The exact shape of the engine's payload isn't finalized and will settle alongside the write API in the next MR, so for now the column is listed in spec/support/shared_examples/models/jsonb_column_validation_todo.yml (which marks the coverage check pending), same as the Govern::Policy jsonb columns. The validator will be added together with the write API.
No development fixtures are added for these tables, same as the sibling tables.
Growth and access patterns
Growth: one row is written per policy evaluation per deployment-gate event, only for organizations opted into the Policies v2 experiment. Nothing writes in this MR and the future callers are all behind the experiment gate, so I expect the volume to stay small (under 1M rows) over the first 3-6 months. Audit mode, which logs every evaluation, is the main growth driver. I'm planning a retention policy (and time-based partitioning if that turns out to be needed) as a follow-up before any wider rollout. But the tables are new and empty right now, which is the cheapest time to change that approach, so let me know what you think.
Writes: a single-row insert per evaluation, plus zero or more violation inserts, all from one internal API call added in the follow-up MR. No updates are expected, the rows are append-only. Deletes happen via CASCADE when a policy is deleted, and via loose foreign key cleanup when an organization is deleted.
Reads: per-policy listings and time-window aggregates use (govern_policy_id, evaluated_at), org-wide listing uses (organization_id, evaluated_at), and project-scoped filtering uses the project_id index. No queries ship in this MR. The read APIs land in the follow-up MR, and any additional filter indexes (for environment_id or user_id) will be added together with those queries once they're known.
There are no new or modified application queries in this MR, so there are no query plans to attach. The DDL itself is exercised by the migration testing pipeline.
Database
Three migrations, targeting milestone %19.4: (1) create govern_policy_evaluations, with its foreign key to govern_policies added inline; (2) create govern_policy_violations, with its foreign key to govern_policy_evaluations added inline; (3) add the govern_policy_violations to govern_policies foreign key in its own migration, using add_concurrent_foreign_key. This follows the convention for a new table that needs two foreign keys, so each transaction locks only one referenced table.
All three migrations are reversible. I've verified up and down locally on all databases.
Migration up
main: == [advisory_lock_connection] object_id: 160000, pg_backend_pid: 9451
main: == 20260819104300 CreateGovernPolicyEvaluations: migrating ====================
main: -- create_table(:govern_policy_evaluations)
main: -> 0.0402s
main: == 20260819104300 CreateGovernPolicyEvaluations: migrated (0.0459s) ===========
main: == 20260819104305 CreateGovernPolicyViolations: migrating =====================
main: -- create_table(:govern_policy_violations)
main: -> 0.0054s
main: == 20260819104305 CreateGovernPolicyViolations: migrated (0.0094s) ============
main: == 20260819104310 AddGovernPoliciesFkToGovernPolicyViolations: migrating ======
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- execute("LOCK TABLE govern_policies, govern_policy_violations IN SHARE ROW EXCLUSIVE MODE")
main: -> 0.0003s
main: -- execute("ALTER TABLE govern_policy_violations ADD CONSTRAINT fk_71f0ac856a FOREIGN KEY (govern_policy_id) REFERENCES govern_policies (id) ON DELETE CASCADE NOT VALID;")
main: -> 0.0004s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0002s
main: -- execute("ALTER TABLE govern_policy_violations VALIDATE CONSTRAINT fk_71f0ac856a;")
main: -> 0.0009s
main: -- execute("RESET statement_timeout")
main: -> 0.0002s
main: == 20260819104310 AddGovernPoliciesFkToGovernPolicyViolations: migrated (0.0133s)
main: == [advisory_lock_connection] object_id: 160000, pg_backend_pid: 9451Migration down
main: == [advisory_lock_connection] object_id: 160200, pg_backend_pid: 9298
main: == 20260819104310 AddGovernPoliciesFkToGovernPolicyViolations: reverting ======
main: -- transaction_open?(nil)
main: -> 0.0000s
main: -- remove_foreign_key(:govern_policy_violations, {:column=>:govern_policy_id})
main: -> 0.0022s
main: == 20260819104310 AddGovernPoliciesFkToGovernPolicyViolations: reverted (0.0506s)
main: == [advisory_lock_connection] object_id: 160200, pg_backend_pid: 9298
main: == [advisory_lock_connection] object_id: 159740, pg_backend_pid: 9363
main: == 20260819104305 CreateGovernPolicyViolations: reverting =====================
main: -- drop_table(:govern_policy_violations)
main: -> 0.0336s
main: == 20260819104305 CreateGovernPolicyViolations: reverted (0.0405s) ============
main: == [advisory_lock_connection] object_id: 159740, pg_backend_pid: 9363
main: == [advisory_lock_connection] object_id: 159740, pg_backend_pid: 9405
main: == 20260819104300 CreateGovernPolicyEvaluations: reverting ====================
main: -- drop_table(:govern_policy_evaluations)
main: -> 0.0349s
main: == 20260819104300 CreateGovernPolicyEvaluations: reverted (0.0401s) ===========
main: == [advisory_lock_connection] object_id: 159740, pg_backend_pid: 9405Both tables live in the gitlab_sec schema, are sharded by organization_id, have table_size: small, and use feature category security_policy_management. Both are registered in config/organizations/transfer_support.yml with tracking issues for organization transfer support (#619206 and #619207), same as the sibling govern tables (#616253, #616254).
Govern::PolicyEvaluation is also added to Organizations::Transfer::UsersService.skipped_transfer_models. The users transfer service auto-discovers models with a single User association and a single organization association, and would otherwise rewrite organization_id on evaluations when their principal user moves organizations. But the evaluation's organization_id derives from its policy, not from the user, so a user transfer must not move these rows.
MR acceptance checklist
Evaluated against the MR acceptance checklist.
How to set up and validate locally
This can be validated in a Rails console by creating a policy, an evaluation, and a violation:
organization = Organizations::Organization.first
policy = Govern::Policy.create!(
organization: organization,
name: "example-policy",
trigger_type: :deployment_requested
)
evaluation = Govern::PolicyEvaluation.create!(
organization: organization,
policy: policy,
trigger_type: :deployment_requested,
mode: :enforce,
verdict: :deny,
policy_version: 1,
evaluated_at: Time.current
)
violation = Govern::PolicyViolation.create!(
organization: organization,
evaluation: evaluation,
policy: policy
)This confirms the associations and constraints work end to end (Govern::Policy#evaluations, Govern::Policy#violations).