Compile policy rules to Rego on create and update

Marcos Rocharequested to merge
mc_rocha-add-rego-rule-transpiler into master

What does this MR do and why?

Compiles a policy's rules to Rego on create and on update. Gitlab::PolicyStore::RuleTranspiler turns one authored rule, a { type, value } jsonb entry, into one self-contained Rego program, stored under rego on the entry that produced it. The entry points are Ports::PolicyRepository#with_compiled_rules on create and #with_updated_rules on update. It reuses ScopeTranspiler's shape and JsonValue.deep_stringify, so there is little new to read, and a rule reads the same whether its keys are the strings a jsonb column returns or the symbols a console session types.

environment is the only emitter here, since a custom rule is passed through as authored, and calendar follows in a separate MR, because normalising its timestamps is a large enough concern to review on its own. Gitlab::PolicyStore::Rules::ALL still advertises calendar as authorable, and the REST endpoint serves that catalogue, so until the follow-up lands, a calendar rule is refused with rule 0: unsupported rule type "calendar". What does not ship in either MR is the validator port, since save-time validation needs a GLAZ validate that is not on master yet.

Compiling on create is also what makes three previously valid policies fail to save, all deliberately: a scan_finding rule, which belongs to a merge request trigger the store does not have; a non-array rules such as { "rules" => [...] }, which would leave every rule with no program; and a custom rule declaring a package other than governance, which the Policy Engine queries and finds nothing in. The alternative in each case is a policy that saves, reads as enforcing in the UI, and enforces nothing.

Two encoding failures used to escape as an unhandled exception rather than a ValidationError, which is a 500 where every other refusal in this list is a 400, since CreateService rescues only ValidationError. A name or tier whose bytes cannot reach UTF-8, and a custom program that is not UTF-8, now fail with a ValidationError naming the value and the encoding found.

A refusal also names an offending value only up to 64 characters, and appends the value's real length. Nothing bounds the size of an authored rule, since rules has no entry in TEXT_LIMITS and there is no database limit for it to mirror, and the message becomes an API response body and a log line. A 200,000-character rule type used to produce a 200,032-character ValidationError message, and now produces one of 116 characters. Nor does a refusal render the value it names, because Integer#to_s on a wide enough value is superlinear: at five million digits, refusing the rule went from 0.280 seconds to 0.000478, against 0.288 seconds to render that Integer once. rule_index is coerced for a different reason, that it is emitted into the generated program and a caller outside the store supplies it.

Design decisions

  • A rename does not touch a compiled rule, unlike a compiled scope. with_updated_scope regenerates scope_rego on a rename, because ScopeTranspiler emits the policy name into it. with_updated_rules does not, because RuleTranspiler never sees the policy name. After update(id, name: "Renamed"), rules is byte-identical and scope_rego is regenerated.
  • One program per rule, not per policy. Each entry's rego carries its own package line, which is why the compiled Rego rides on the entry rather than in a column of its own. A combined per-policy module stays reachable from that: keep the first package governance line, strip it from the rest. Combining in the API serialization layer is under discussion, since evaluating one module per policy is measurably cheaper, so an emitted violation carries its rule_index to keep attribution through a merge. violation is a set, and two rules emitting identical objects would otherwise deduplicate into one. A policy still fires when any one of its rules fires, but that OR belongs to whoever evaluates the programs.
  • Output is package governance exposing violation contains {"msg", "details"}, one of the three shapes the Policy Engine parses, following the reference rules linked below. No import rego.v1 is emitted: Regorus 0.11 enables it by default, so the line is a verified no-op.
  • A name conflict is reported before anything compiles. validate_name_available! moved ahead of with_compiled_scope and with_compiled_rules on both write paths, so a call that both takes a name already in use and carries an uncompilable rule reports the name: the cheaper check, and the one the caller can act on without reading the rest.

The full reasoning for each of those is in the commit message.

How to set up and validate locally

Requires an Ultimate licence. Every block below runs in gdk rails console.

  1. Turn the experiment on and confirm it took effect, because until it is active every step below returns Policy Store experiment is not active for this organization and no rule reaches the transpiler
Feature.enable(:security_policies_v2)
ApplicationSetting.current.update!(policy_store_experiment_enabled: true)

organization = Organizations::Organization.first
organization.policy_store_experiment_active?
# => true
  1. Define the helper the rest of the steps call. It reads the message rather than the payload when a create fails, since a rejected policy has no payload, and it returns the stored rule so the next step can look at the whole entry
def create_rule(organization, rules, name:)
  result = Security::SecurityOrchestrationPolicies::PolicyStore::CreateService.new(
    organization: organization,
    params: { name: name, trigger_type: "deployment_requested", rules: rules }
  ).execute

  return puts(result.message) unless result.success?

  result.payload[:policy].rules.first
end
  1. Compile an environment rule. Verify the program carries both conditions, because authoring a name and a tier together requires both to hold rather than either
environment_rule = { "type" => "environment",
                     "value" => { "names" => ["prod-us-east"], "tiers" => ["production"] } }

stored_environment = create_rule(organization, [environment_rule], name: "Production only")

puts stored_environment["rego"] if stored_environment
  1. Resend a rule through update with a hand-written rego, deliberately supplying "package attacker". Verify the stored program is package governance matching on {"staging"} rather than the attacker package sent, because an authored rego is derived and does not survive an update either, and before this MR that call stored package attacker verbatim
create_rule(organization, [{ "type" => "environment", "value" => { "tiers" => ["production"] } }],
  name: "Compiled on update")

policy = Gitlab::PolicyStore.list(organization_id: organization.id)
  .find { |candidate| candidate.name == "Compiled on update" }

authored = [{ "type" => "environment", "value" => { "tiers" => ["staging"] }, "rego" => "package attacker" }]

puts(policy ? Gitlab::PolicyStore.update(policy.id, rules: authored).rules.first["rego"] : "create failed, see the message above")
  1. Author a custom rule declaring package governance. Verify the stored rego is byte-identical to the value authored, because a custom rule's value is already a whole program, so it is stored as authored rather than reformatted
program = "package governance\n\nviolation contains {\"msg\": \"no production deploys\"}\n"

stored_custom = create_rule(organization, [{ "type" => "custom", "value" => program }],
  name: "Hand-written program")

puts(stored_custom ? stored_custom["rego"] == program : "create failed, see the message above")
# => true
  1. Author a custom rule declaring the wrong package. Verify it fails with rule 0: custom rule must declare `package governance`, found "gitlab.policy", because the Policy Engine queries one package, so a program declaring another comes back empty with no error, which nothing downstream can tell apart from a rule that did not fire
create_rule(organization, [{ "type" => "custom", "value" => "package gitlab.policy\n\nallow := true\n" }],
  name: "Wrong package")
  1. Author the two rule types this MR has no emitter for: scan_finding, which the store accepted before this MR, and calendar, whose emitter lands in the follow-up. Verify each fails with rule 0: unsupported rule type, naming the type, and that neither stored anything, because a rule with no emitter would save a policy that reads as enforcing and enforces nothing
before = Gitlab::PolicyStore.list(organization_id: organization.id).size

create_rule(organization, [{ "type" => "scan_finding" }], name: "Scan finding policy")
create_rule(organization, [{ "type" => "calendar", "value" => {} }], name: "Calendar policy")

Gitlab::PolicyStore.list(organization_id: organization.id).size == before
# => true
  1. Confirm the opposite path still saves. Create a policy with no rules at all and verify rules comes back [], because having nothing to compile is not the same as having nothing to store
result = Security::SecurityOrchestrationPolicies::PolicyStore::CreateService.new(
  organization: organization,
  params: { name: "No rules at all", trigger_type: "deployment_requested", rules: [] }
).execute

puts(result.success? ? result.payload[:policy].rules.inspect : result.message)
# => []

Expected result: before this MR, a stored rule was the { type, value } entry as authored, a scan_finding rule saved, and an update stored whatever rego it was handed. After it, an environment or custom entry carries a third key holding a compiled package governance program, on create and on update alike, and a rule the store cannot compile, including a calendar rule, fails either call with the rule's index in the message. A policy with no rules is unaffected.

Names are unique within an organization, so re-running any step needs a fresh name:. The helper prints the error, so a repeat shows as a message rather than an exception.

References

Edited by Marcos Rocha

Merge request reports