Draft: Policy Store v2 deployment-gate PoC[ci skip]

Marcos Rocharequested to merge
mc_rocha_policy_store_poc into master

What & why

Ports the deployment-gate PoC into gitlab-org/gitlab on the Policy Store v2 data model, per .agents/output/policy-store-v2-migration-implementation-plan.md. Backend-only, no UI, gated behind the deployment_gate_policy_store_v2 feature flag (default off).

[ci skip] on the commit for now — opening as Draft for review.

Architecture (two gems + monolith seam)

  • gems/gitlab-policy-engine — ported from the fork (gate / evaluator / rego + typed-rule evaluation, embedded OPA Go FFI). New ScopeEvaluation sibling to RegoEvaluation. The engine's rules-evaluation graph is untouched.
  • gems/gitlab-security-policy-store — ActiveRecord-free. ScopeTranspiler is a byte-exact Ruby port of the gitlab-policy-to-rego PoC; Store#load_policies returns unfiltered POROs with scope_rego.
  • Monolith — policies table in the sec schema (gitlab_sec, SecApplicationRecord) + Security::Policies::V2::Policy (per-column JSON-schema validation, before_save scope compilation), StoreAdapter, ContextBuilders::Scope, and PolicyLookup (the only place that talks to both gems: builds scope context → filters via ScopeEvaluation → presents to the engine). Deployment gate lands in EE::Ci::ProcessBuildService; ci_build_id + security_policy_id columns added to protected_environment_approval_rules (no real FK — security_policy_id is cross-schema to gitlab_sec, so a loose foreign key nullifies it when the policy is deleted); new security_policy_denied failure reason; security_policies:v2:seed rake task for authoring.

Two decisions to sanity-check in review

  1. ScopeEvaluation (Approach A): the OPA binding hardcodes rego.Query("data.policy") — the plan's query: arg doesn't exist — so it rewrites the stored package gitlab.scope → package policy at eval time (fail-closed). Stored scope_rego stays faithful to the GOVERN-006 contract.
  2. security_policy_id provenance: CheckResult carries no policy link and the engine must stay untouched, so PolicyLookup threads security_policy_id into each action hash. Populates the approval-rule FK for the typed-rules path (deferred for rego-authored actions).

Test coverage (local)

  • store gem 20/20, engine scope specs 10/10
  • monolith model / adapter / context-builder / PolicyLookup unit specs
  • end-to-end A–E deployment-gate integration spec (no stubs on the store/engine/transpiler seam) + existing process_build_service_spec regression 13/13

How the policy store works

The store gem answers "which policies exist for this trigger, and here's their compiled scope" — nothing more. It compiles scope to Rego on write (ScopeTranspiler) and returns immutable, unfiltered policy value objects on read (Store). The monolith adapter then uses the engine gem to decide which ones actually apply and what to do.

Diagrams

Storage & schema (sec database + loose FK)

The policies table lives in the sec database (gitlab_sec), so it cannot hold a real foreign key from protected_environment_approval_rules (gitlab_main_org). A loose foreign key covers the link instead: a DELETE tracking trigger on policies feeds the loose-FK worker, which nullifies security_policy_id.

flowchart LR
    subgraph SEC["🔒 sec database · gitlab_sec"]
        POL[("policies<br/>Security::Policies::V2::Policy<br/>(SecApplicationRecord)")]
    end
    subgraph MAIN["🏛️ main database · gitlab_main_org"]
        PEAR[("protected_environment_approval_rules<br/>security_policy_id · nullable")]
    end
    PEAR -. "loose FK · on_delete: async_nullify" .-> POL
    TRIG["DELETE on policies → tracking trigger →<br/>loose_foreign_keys_deleted_records →<br/>LFK worker nullifies security_policy_id"]
    POL -.-> TRIG

Data flow

flowchart TD
    A["EE::Ci::ProcessBuildService#process<br/>(deployment job, on 'success')"] --> B{should_block_processable?}
    B --> C["deployment_gate_blocks?<br/>FF: deployment_gate_policy_store_v2"]
    C --> D["Gate.for(:deployment,:requested)<br/>.enforce(project, environment, user, entity)"]
    D --> E["Evaluator#enforce<br/>(context + evaluate hooks)"]
    E --> F["PolicyLookup#call<br/>— monolith seam, both gems —"]
    F --> G["ContextBuilders::Scope.build<br/>project → {project, groups,<br/>compliance_frameworks, security_attributes}"]
    F --> H["StoreAdapter → Store#load_policies<br/>(active_for_trigger, UNFILTERED)"]
    H -.reads.-> DB[("policies<br/>gitlab_sec")]
    G --> I{"ScopeEvaluation#applies?<br/>rewrite 'package gitlab.scope'→'package policy'<br/>FAIL-CLOSED (error ⇒ in-scope)"}
    H --> I
    I -.OPA FFI.-> OPA[["embedded OPA<br/>data.policy"]]
    I --> J["present_for_evaluator → EnginePolicy<br/>(+ security_policy_id merged into each action)"]
    J --> K["Evaluator: RegoEvaluation / TypedRuleEvaluation<br/>⇒ CheckResult(actions, reasons)"]
    K --> L["enforce_deployment_actions<br/>(per action[:type])"]
    L --> M["'deny' → DenyOperation::Deployment<br/>build.drop!(:security_policy_denied)"]
    L --> N["'require_approval' → ApprovalRule::Deployment<br/>find_or_create_for_build!(ephemeral rule)"]
    N -.writes.-> DB2[("protected_environment_<br/>approval_rules")]
    K --> O{"CheckResult#matched?"}
    O -->|"true & !failed?"| P["actionize: build.when = 'manual'"]

Policy Store class diagram

classDiagram
    class Store {
        +initialize(adapter:)
        +load_policies(trigger_id:) Array~Policy~
        -present(row) Policy
    }
    class Policy {
        <<Data value object, immutable>>
        +id
        +name
        +trigger_id
        +scope_rego
        +rules
        +actions
        +mode
        +organization_id
        +version
    }
    class ScopeTranspiler {
        <<module_function, pure>>
        +transpile(policy_scope, policy_name:) String
        -build_scope(raw) IR
        -generate_scope_rego(policies) String
    }
    class Adapter {
        <<injected port>>
        +call(trigger_id:) Enumerable~row~
    }

    Store --> Policy : maps rows to
    Store ..> Adapter : calls (dependency injection)
    note for Store "Returns every trigger-matching policy\nUNFILTERED — does not resolve scope.\nscope_rego is handed back, not acted on."
    note for ScopeTranspiler "Port of the gitlab-policy-to-rego PoC.\nOutput package: gitlab.scope"

Gems integration

flowchart TB
    subgraph MONO["🏛️ GitLab monolith (EE)"]
        direction TB
        PB["EE::Ci::ProcessBuildService\n#deployment_gate_blocks?"]
        PL["Adapters::PolicyLookup\n(the cross-gem seam)"]
        SA["V2::StoreAdapter\n#call(trigger_id:)"]
        SC["ContextBuilders::Scope\n(project → JSON input)"]
        MODEL["Security::Policies::V2::Policy\n(policies table · gitlab_sec)"]
        EXEC["Executors::\nDenyOperation / ApprovalRule"]
    end

    subgraph STORE["📦 gitlab-security-policy-store"]
        direction TB
        ST["Store#load_policies"]
        TR["ScopeTranspiler.transpile"]
        POL["Policy (value object)"]
    end

    subgraph ENGINE["📦 gitlab-policy-engine"]
        direction TB
        GATE["Gate.for(:deployment,:requested)"]
        SE["ScopeEvaluation#applies?"]
        EV["Evaluator (rules → actions)"]
        OPA["Gitlab::Opa::Engine\n(FFI → Go/OPA Rego)"]
    end

    PB -->|enforce| GATE
    GATE -->|policy_lookup port| PL
    PL -->|load_policies| ST
    ST -->|adapter.call| SA
    SA --> MODEL
    ST --> POL
    POL -->|"candidates (unfiltered)"| PL
    PL -->|build| SC
    PL -->|"scope_rego + context"| SE
    SE --> OPA
    PL -->|"in-scope survivors"| EV
    EV -->|rules eval| OPA
    EV -->|actions| GATE
    GATE --> PB
    PB --> EXEC

    MODEL -.->|"before_save: compile"| TR
    TR -.->|scope_rego| MODEL

    classDef gem fill:#e8f0ff,stroke:#3b6,stroke-width:1px
    classDef mono fill:#fff5e6,stroke:#e90,stroke-width:1px
    class STORE,ENGINE gem
    class MONO mono

The cross-gem seam

flowchart TB
    subgraph SEAM["Adapters::PolicyLookup#call — the ONLY place both gems meet"]
        direction TB
        A1["1 · load_candidates(trigger)\n→ store.load_policies"]
        A2["2 · ContextBuilders::Scope.new(project).build"]
        A3["3 · .select { in_scope? }\n→ ScopeEvaluation#applies?"]
        A4["4 · .map { present_for_evaluator }\n→ EnginePolicy value objects\n(+ security_policy_id provenance)"]
        A1 --> A3
        A2 --> A3
        A3 --> A4
    end

    STORE["📦 store gem\nowns: WHAT policies exist\n(persistence + scope compilation)"]
    ENGINE["📦 engine gem\nowns: DOES it apply / WHAT to do\n(scope eval + rules → actions)"]

    STORE -->|candidates + scope_rego| A1
    A3 -->|delegates scope eval| ENGINE
    A4 -->|feeds| ENGINE

    note1["Neither gem imports the other.\nThe monolith adapter wires them together,\nkeeping both gems ActiveRecord-free and\nextractable to standalone repos later."]
    SEAM -.-> note1

Convert scope to rego

sequenceDiagram
    autonumber
    actor Author
    participant M as V2::Policy (AR model)
    participant T as ScopeTranspiler
    participant DB as policies

    Author->>M: create / update (policy_scope: {...})
    activate M
    M->>M: recompile_scope_rego? (new_record? || policy_scope_changed?)
    alt scope changed or new
        M->>T: transpile(policy_scope, policy_name: name)
        activate T
        T->>T: build_scope → IR (normalize, detect empty)
        T->>T: generate_scope_rego (string templating)
        T-->>M: scope_rego  ("package gitlab.scope ...")
        deactivate T
        M->>M: self.scope_rego = <result>
    end
    M->>DB: INSERT/UPDATE (policy_scope + scope_rego together)
    deactivate M
    Note over M,DB: policy_scope stays the source of truth;\nscope_rego is a derived, cached compilation.
flowchart LR
    IN["policy_scope hash\n{projects, groups,\ncompliance_frameworks,\nbusiness_impact, ...}"]
    --> BUILD["build_scope\n(port of ir.ts)"]
    BUILD --> IR["Normalized IR\n• match_mode all/any\n• ids() coercion\n• include/exclude dims\n• empty? flag"]
    IR --> GEN["generate_scope_block\n(port of generate.ts)"]

    GEN --> EMPTY{"scope empty?"}
    EMPTY -->|yes| R1["results { applies: true,\n'no policy_scope' }"]
    EMPTY -->|no| R2["scope_excluded_i rules\n+ scope_included_i rules\n+ scope_applies_i =\n  not excluded AND included"]

    R1 --> OUT["scope_rego\npackage gitlab.scope\n+ SCOPE_PRELUDE\n(applicable / not_applicable\n / applicability)"]
    R2 --> OUT

Runtime path

sequenceDiagram
    autonumber
    participant PBS as ProcessBuildService
    participant Gate as PolicyEngine::Gate
    participant PL as PolicyLookup (seam)
    participant Store as Store
    participant SA as StoreAdapter
    participant DB as policies
    participant Scope as ContextBuilders::Scope
    participant SE as ScopeEvaluation
    participant OPA as Opa::Engine (FFI/Go)
    participant EV as Evaluator
    participant Ex as Executors

    PBS->>Gate: enforce(project, environment, user, entity)
    Gate->>PL: call(project:, trigger: :deployment_requested)
    Note over PL: return [] unless trigger supported (PoC: deployment only)

    PL->>Store: load_policies(trigger_id:)
    Store->>SA: adapter.call(trigger_id:)
    SA->>DB: Policy.active_for_trigger(trigger_id)
    DB-->>SA: rows (active, this trigger)
    SA-->>Store: rows
    Store-->>PL: [Policy value objects]  (UNFILTERED, scope_rego incl.)

    PL->>Scope: build(project)
    Scope-->>PL: scope context JSON\n{project, groups, compliance_frameworks, security_attributes}

    loop each candidate policy
        PL->>SE: applies?(scope_rego, context)
        SE->>OPA: evaluate(package policy, input: context)
        OPA-->>SE: applicability.results
        SE-->>PL: true / false  (fails closed → true)
    end

    PL-->>Gate: in-scope policies mapped to EnginePolicy
    Gate->>EV: evaluate rules for survivors
    EV->>OPA: evaluate typed-rules / rego
    EV-->>Gate: actions (deny / require_approval)
    Gate-->>PBS: check_result (matched?, actions, reasons)

    alt actions present
        PBS->>Ex: DenyOperation / ApprovalRule .execute
    end
    PBS->>PBS: block / actionize build if matched?

How to test locally

All backend/console — no browser needed (watch the pipeline UI at the end if you like). The five cases below are the same ones the automated integration spec covers.

One-time setup

# 1. Build the embedded OPA shared library (Go >= 1.22 required)
cd gems/gitlab-policy-engine/ext && make       # or: bundle exec rake gitlab:opa:compile
bundle exec rake gitlab:opa:check              # => "OPA shared library is loaded"

# 2. Install the two new path gems + run migrations
bundle install
bundle exec rails db:migrate
# rails console
Feature.enable(:deployment_gate_policy_store_v2)

project = Project.find_by_full_path("your-group/your-project")
user    = project.first_owner

# Protect "production" and give the user deploy access (so the build isn't
# dropped as protected_environment_failure before the gate runs).
project.protected_environments.create!(
  name: "production",
  deploy_access_levels_attributes: [{ user_id: user.id }]
)

Seed a policy (the no-UI authoring path)

# require_approval, applies everywhere:
bundle exec rake "security_policies:v2:seed[your-group/your-project]"
# deny:
bundle exec rake "security_policies:v2:seed[your-group/your-project,deny]"
# scoped to a DIFFERENT project id (proves scope_rego is evaluated):
bundle exec rake "security_policies:v2:seed[your-group/your-project,require_approval,999999]"

The task prints the generated scope_rego (starts with package gitlab.scope), confirming the transpiler ran on save. Or create the row directly: Security::Policies::V2::Policy.create!(...).

Trigger the gate

Add a production deployment job to the project's .gitlab-ci.yml and run a pipeline (push, or Ci::CreatePipelineService):

deploy-prod:
  stage: deploy
  environment: { name: production }
  script: echo deploying

Expected results

Case Policy deploy-prod job Approval rule
A require_approval blocked → manual one ProtectedEnvironments::ApprovalRule with ci_build_id + security_policy_id
B deny failed, reason security_policy_denied, non-retryable —
C any (broken lookup) fail-closed → security_policy_denied —
D none / flag off proceeds normally (pending) —
E scoped to another project no-op (proves scope filtering) — 
pipeline = Ci::Pipeline.find_by_project_id(project.id)
build = pipeline.builds.find { |b| b.persisted_environment&.name == "production" }
build.reload.status        # "manual" (A) / "failed" (B, C)
build.failure_reason       # "security_policy_denied" for deny / fail-closed
build.retryable?           # false for security_policy_denied
ProtectedEnvironments::ApprovalRule.for_build(build)  # case A

Screenshots

Screenshot_2026-07-14_at_4.51.16_PM

Automated (deterministic, no stubs on the store/engine/transpiler seam)

bundle exec rspec ee/spec/services/security/policies/deployment_gate_integration_spec.rb
cd gems/gitlab-security-policy-store && bundle exec rspec
cd gems/gitlab-policy-engine && bundle exec rspec spec/gitlab/policy_engine/scope_evaluation_spec.rb

Deferred (PoC scope)

Scope cache (roaring-bitmap), the non-deployment triggers + their context builders, rego-path security_policy_id, and decomposition into the standalone policy-store repo.

🤖 Generated with Claude Code

Edited by Marcos Rocha

Merge request reports