Add MalwareAdvisoriesFinder for Dependency Firewall malicious enforcement

Bala Kumarrequested to merge
22692-malware-advisories-finder-poc into master

What & why

Adds the finder that lets the Dependency Firewall (DFW) detect malicious packages inside the monolith — the data-query layer for the malicious signal.

Security::DependencyFirewall::FetchPackageMaliciousService (added in !242988 (merged)) is the enforcement entry point, mirroring FetchPackageVulnerabilitiesService. It delegates to PackageMetadata::MalwareAdvisoriesFinder, which !245899 (merged) introduced as a flag-gated no-op (returns []).

!245899 (merged) has since merged, so the flag and the no-op finder are now on master. This MR targets master and replaces that no-op with the real malware-advisory query, plus the index that serves it.

This is the Rails path that replaces the superseded "Malicious Packages API" direction (a standalone external service): the DFW is part of the monolith, so enforcement is done in-process through a finder. See work item gitlab-org#22692.

What this MR does

  • PackageMetadata::MalwareAdvisoriesFinder (ee/app/finders/package_metadata/malware_advisories_finder.rb) — replaces the flag-gated no-op (introduced by !245899 (merged)) with the real query. Takes (name:, purl_type:, version:) (matching FetchPackageMaliciousService's signature) and returns the PackageMetadata::MalwareAffectedPackage records (with their MalwareAdvisory eager-loaded) that flag the component and whose affected range includes version. The version is matched client-side via SemverDialects (Gitlab::VulnerabilityScanning::AdvisoryUtils#build_matcher), exactly as FetchPackageVulnerabilitiesService does, because malware affected ranges are arbitrary version disjunctions rather than SQL-comparable ranges. A blank version matches all ranges (fail-closed). The caller shapes each advisory into its audit/return contract.
  • Query scopes on PackageMetadata::MalwareAffectedPackage — for_occurrences / with_advisory, mirroring PackageMetadata::AffectedPackage.
  • Index migration — i_pm_malware_affected_packages_on_purl_type_and_package_name on (purl_type, package_name). The table previously only had i_pm_malware_affected_packages_unique_for_upsert (leading with pm_malware_advisory_id), which cannot serve this lookup. The abbreviated i_ prefix is used because the full index_... name would be 64 chars, over Postgres's 63-char identifier limit.
  • Flag-gated (flag lives in !245899 (merged)) — the finder's DB lookup stays behind dependency_firewall_malicious_packages (the wip, default-off, instance-wide flag defined in !245899 (merged)). This MR does not touch the flag definition; it fills in the query the flag guards. With the flag off, #execute returns [] before querying, so the query can be enabled for performance testing and switched off if it causes production issues.
  • Specs — finder spec + model scope specs (see Testing).

Because the finder does client-side version filtering, #execute returns an Array rather than an ActiveRecord::Relation, so it is added to spec/support/finder_collection_allowlist.yml with a reason (same rationale as the vulnerabilities equivalent being a service).

Database review

The malware tables live on the gitlab_sec_cell_local schema (the sec database).

Migration

Regular schema migration (db/migrate) using add_concurrent_index + disable_ddl_transaction!. Milestone 19.3. It is a regular (not post-deployment) migration because pm_malware_affected_packages is a newly-introduced table that is currently empty, so the concurrent index builds instantly with no locking concern, and the index ships alongside the application code that uses it. Reversible via explicit up/down.

INDEX_NAME = 'i_pm_malware_affected_packages_on_purl_type_and_package_name'

def up
  add_concurrent_index :pm_malware_affected_packages, [:purl_type, :package_name], name: INDEX_NAME
end

def down
  remove_concurrent_index_by_name :pm_malware_affected_packages, INDEX_NAME
end

DDL added to the schema:

CREATE INDEX i_pm_malware_affected_packages_on_purl_type_and_package_name
  ON pm_malware_affected_packages USING btree (purl_type, package_name);
Migration output (up / down) 
# up
sec: == 20260707120000 AddPurlTypePackageNameIndexToPmMalwareAffectedPackages: migrating
sec: -- index_exists?(:pm_malware_affected_packages, [:purl_type, :package_name], {:name=>"i_pm_malware_affected_packages_on_purl_type_and_package_name", :algorithm=>:concurrently})
sec: -- add_index(:pm_malware_affected_packages, [:purl_type, :package_name], {:name=>"i_pm_malware_affected_packages_on_purl_type_and_package_name", :algorithm=>:concurrently})
sec: == 20260707120000 AddPurlTypePackageNameIndexToPmMalwareAffectedPackages: migrated (0.0773s)

# down
sec: == 20260707120000 AddPurlTypePackageNameIndexToPmMalwareAffectedPackages: reverting
sec: -- index_name_exists?(:pm_malware_affected_packages, "i_pm_malware_affected_packages_on_purl_type_and_package_name")
sec: -- remove_index(:pm_malware_affected_packages, {:algorithm=>:concurrently, :name=>"i_pm_malware_affected_packages_on_purl_type_and_package_name"})
sec: == 20260707120000 AddPurlTypePackageNameIndexToPmMalwareAffectedPackages: reverted (0.1239s)

Table size and growth

pm_malware_affected_packages is a new table, currently empty on GitLab.com; it is populated by the malware-advisory ingestion feed (separate work). Each malware advisory enumerates its affected package versions as rows keyed by (purl_type, package_name). Expected order of magnitude is comparable to (well below) the public pm_affected_packages table, which carries the analogous index_pm_affected_packages_on_purl_type_and_package_name index for the same access pattern.

Query and plans

When dependency_firewall_malicious_packages is enabled, for a single component (e.g. name: "lodash", purl_type: "npm" → integer 6) MalwareAdvisoriesFinder#execute evaluates for_occurrences(...).with_advisory, which issues the lookup below plus a preload of the matched advisories (when the flag is off, #execute returns [] and neither query runs):

-- affected packages for the component (this is where the new index is used)
WITH occurrences_cte(purl_type, name) AS (VALUES (6, 'lodash'))
SELECT DISTINCT "pm_malware_affected_packages".*
FROM "pm_malware_affected_packages"
INNER JOIN occurrences_cte
  ON occurrences_cte.purl_type = pm_malware_affected_packages.purl_type
  AND occurrences_cte.name = pm_malware_affected_packages.package_name;

-- advisories preloaded for the matched rows (from includes(:malware_advisory))
SELECT "pm_malware_advisories".*
FROM "pm_malware_advisories"
WHERE "pm_malware_advisories"."id" IN (/* pm_malware_advisory_id values from query 1 */);

The (purl_type, package_name) join predicate is served by the new index. This is a single-component point lookup — one (purl_type, package_name) pair per call, with no namespace/project scope to scan and no fan-out, so cost stays bounded regardless of group size. The version filter is deliberately not in SQL: malware affected_range is an arbitrary explicit-version disjunction (=1.0.0 || =1.0.1 || …), not a SQL-comparable range, so version_matches? applies it in Ruby via SemverDialects over the (small) result set — the same approach as FetchPackageVulnerabilitiesService.

The plans below are from a local GDK sec DB seeded with 60,001 affected-package rows across 3,000 advisories (synthetic), inside a rolled-back transaction. They demonstrate the index is chosen and the improvement over the pre-existing unique index; production-scale plans from Database Lab will be attached from the db:gitlabcom-database-testing run before review.

EXPLAIN (ANALYZE, BUFFERS) — with vs. without the new index 
########## WITH new index (this MR) ##########
Unique  (cost=2.44..2.46 rows=1 width=47) (actual time=0.009..0.009 rows=1 loops=1)
  Buffers: shared hit=4
  ->  Sort  (cost=2.44..2.45 rows=1 width=47) (actual time=0.009..0.009 rows=1 loops=1)
        ->  Index Scan using i_pm_malware_affected_packages_on_purl_type_and_package_name on pm_malware_affected_packages  (cost=0.41..2.43 rows=1 width=47) (actual time=0.005..0.005 rows=1 loops=1)
              Index Cond: ((purl_type = 6) AND (package_name = 'lodash'::text))
              Buffers: shared hit=4
Execution Time: 0.018 ms

########## WITHOUT new index (falls back to i_pm_malware_affected_packages_unique_for_upsert) ##########
Unique  (cost=1001.32..1001.34 rows=1 width=47) (actual time=0.780..0.780 rows=1 loops=1)
  Buffers: shared hit=402
  ->  Sort  (cost=1001.32..1001.32 rows=1 width=47) (actual time=0.779..0.779 rows=1 loops=1)
        ->  Index Scan using i_pm_malware_affected_packages_unique_for_upsert on pm_malware_affected_packages  (cost=0.29..1001.31 rows=1 width=47) (actual time=0.778..0.778 rows=1 loops=1)
              Index Cond: ((purl_type = 6) AND (package_name = 'lodash'::text))
              Buffers: shared hit=402
Execution Time: 0.785 ms

DB Lab query run : https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/53890/commands/156226

Without the new index the planner falls back to the unique upsert index (leading column pm_malware_advisory_id), forcing a full index scan filtered on the non-leading (purl_type, package_name) columns: 402 buffers / 0.785 ms vs. 4 buffers / 0.018 ms with the dedicated index — and that gap grows with table size, since the fallback cost scales with the number of distinct advisories.

Database review checklist

  • db/structure.sql updated + db/schema_migrations/20260707120000 added
  • Migration is reversible (up/down), verified via db:migrate:up:sec / db:migrate:down:sec
  • Index created with add_concurrent_index + disable_ddl_transaction!, explicitly named
  • Raw SQL + query plans included above; application code using the index is in this MR
  • Database Lab plans for the for_occurrences lookup (the WITH occurrences_cte … query above) attached — the same query run without and with the new index (in Joe: run it, CREATE INDEX i_pm_malware_affected_packages_on_purl_type_and_package_name …, run again), via the db:gitlabcom-database-testing job or against the sec database. Note: pm_malware_affected_packages is empty on GitLab.com, so both Lab plans are near-instant and mainly confirm the index builds and is chosen — the seeded local EXPLAIN above is the with/without comparison that shows the improvement at scale.

Feature flag

dependency_firewall_malicious_packages (defined in the now-merged !245899 (merged); type: wip, default off, separate from dependency_firewall_phase1) gates the finder's DB lookup as a query kill-switch, evaluated instance-wide via Feature.enabled?(:dependency_firewall_malicious_packages, :instance). Enable it to performance-test the query; disable it to stop the query immediately if production performance issues appear. Rollout tracked in #605121.

Testing

26 examples, 0 failures across the finder, model, and service specs:

  • ee/spec/finders/package_metadata/malware_advisories_finder_spec.rb — match on (name, purl_type) + in-range version, version out of range, blank version (fail-closed), non-matching name / purl_type, SemverDialects::Error handling, and the flag-off short-circuit.
  • ee/spec/models/package_metadata/malware_affected_package_spec.rb — coverage for the for_occurrences and with_advisory scopes.
  • ee/spec/services/security/dependency_firewall/fetch_package_malicious_service_spec.rb — the service wraps each finder match as { advisory: … } once the finder is real, and returns [] when the flag is off.
  • RuboCop clean on all changed Ruby files.

Local test setup

This MR is the data-query layer; the steps below exercise it end to end in GDK through the merged enforcement path (FetchPackageMaliciousService → MalwareAdvisoriesFinder). Steps 1–3 are prerequisites; 4a tests this MR's finder in isolation, 4b tests it inside a real block.

1. Enable the flags + namespace opt-in (Rails console):

project = Project.find_by_full_path('gitlab-org/.../your-project')
Feature.enable(:dependency_firewall_malicious_packages)        # this MR's query kill-switch (instance-wide)
Feature.enable(:dependency_firewall_phase1, project)           # enforcement gate
project.root_ancestor.namespace_settings.update!(dependency_firewall_enabled: true)

2. Seed a malicious advisory flagging npm lodash (all versions):

adv = PackageMetadata::MalwareAdvisory.create!(
  advisory_xid: 'GLAM-2099-01-00001', source_xid: :glam, published_date: Date.current,
  title: 'Test malicious lodash', description: 'local test',
  identifiers: [{ 'type' => 'GLAM', 'name' => 'GLAM-2099-01-00001',
                  'value' => 'GLAM-2099-01-00001', 'url' => 'https://example.com' }])
PackageMetadata::MalwareAffectedPackage.create!(
  malware_advisory: adv, purl_type: :npm, package_name: 'lodash', affected_range: '>=0')
# advisory_xid must match /\AGLAM-\d{4}-\d{2}-\d{5,}\z/

3. Add a malicious policy to the project's security-policy project .gitlab/security-policies/policy.yml (syncs into project.security_policies):

dependency_firewall_policy:
  - name: Block malicious packages
    enabled: true
    enforcement_type: enforced
    rules:
      - type: malicious
        denied:
          - is_malicious: true

4a. Verify the finder directly (this MR's unit):

PackageMetadata::MalwareAdvisoriesFinder.new(name: 'lodash', purl_type: 'npm', version: '4.17.21').execute
# => [#<PackageMetadata::MalwareAffectedPackage ... advisory GLAM-2099-01-00001>]
PackageMetadata::MalwareAdvisoriesFinder.new(name: 'left-pad', purl_type: 'npm', version: '1.0.0').execute
# => []

4b. Verify the full enforcement path (finder feeding the block):

E = Security::DependencyFirewall::EnforcementService
E.firewall_check(project: project, pkg_type: 'npm', name: 'lodash', version: '4.17.21',
  operation: E::PACKAGE_DOWNLOAD, current_user: User.find_by_username('root'))
# => ServiceResponse(status: :error, reason: :blocked,
#    message: "Package 'lodash' violates 'Block malicious packages' policy")

Kill-switch off → no query runs, package allowed:

Feature.disable(:dependency_firewall_malicious_packages)
PackageMetadata::MalwareAdvisoriesFinder.new(name: 'lodash', purl_type: 'npm', version: '4.17.21').execute  # => []

Screenshots

image

Out of scope

  • Populating the data — the malware-advisory ingestion feed is separate work; until it lands the tables are empty and the finder returns [].
  • Surfacing the advisory in the audit event — the finder returns the advisory record so a follow-up can name what flagged a package; wiring that into the audit payload is not part of this MR.

References

Closes #606161

Edited by Hannah Baker

Merge request reports