Orbit: Add Package Registry entities and edges to the knowledge graph

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Context

The Orbit knowledge graph currently has zero artifact entities. The server-side schema covers ci, code_review, core, plan, security, and source_code domains, but nothing for packages, container images, or deployments.

During the AR agentic spike (#600994) we found that answering "what's the blast radius of @gitlab/ui?" required 20+ REST bridge calls. A single Orbit traversal would suffice if Package existed as a node with a BUILT_BY edge to Pipeline.

Package Registry data lives in the GitLab monolith database (packages_packages, packages_package_files, packages_build_infos, etc.) and is replicated to ClickHouse via Siphon. gkg-indexer is the downstream consumer that materializes the Orbit graph from those ClickHouse tables.

Proposed entities

  • Package — sourced from packages_packages
    • Properties: name, version, package_type (npm, maven, pypi, nuget, golang, cargo, generic, debian, conan, rubygems, helm, terraform_module, rpm, composer, ml_model), status, project_id, created_at, updated_at
  • PackageFile — sourced from packages_package_files
    • Properties: file_name, size, file_sha256, status, created_at

Proposed edges

  • HAS_PACKAGE — Project → Package
    • Source: packages_packages.project_id
    • Impact: basic package-to-project ownership
  • HAS_FILE — Package → PackageFile
    • Source: packages_package_files.package_id
    • Impact: package contents
  • BUILT_BY (package-level) — Package → Pipeline
    • Source: packages_build_infos(package_id, pipeline_id) join table
    • Impact: package provenance ("which pipeline built this artifact")
  • BUILT_BY (file-level) — PackageFile → Pipeline
    • Source: packages_package_file_build_infos(package_file_id, pipeline_id) join table
    • Impact: per-artifact provenance for SBOM-style queries
  • DECLARES_DEPENDENCY — Package → (name, version_pattern)
    • Source: packages_dependency_links + packages_dependencies
    • Impact: declared external deps; populated today by npm, maven, nuget, composer
  • CONSUMED_BY — Package → Project
    • Source: not derivable from current tables — requires new ingestion (see Deferred work)
    • Impact: highest-impact edge for AR. Enables blast-radius queries in a single traversal
  • AFFECTED_BY — Package → Vulnerability
    • Source: security_findings with location.dependency matched on (package_name, package_type)
    • Impact: package-to-vulnerability linkage

Plan

Phase 1 — Minimum viable: provenance edge

Target LOE: S (days–1 week)

Ship Package + HAS_PACKAGE + BUILT_BY (package-level). This is the smallest slice that lets agents answer "which pipeline built this package?" — the spike's primary unmet question. Package-level BUILT_BY (via packages_build_infos) is preferred over file-level for v1 because it's a single, narrower edge with smaller fan-out per package.

Monolith deliverables — generate Siphon tables via the documented Rake task (ClickHouse table design with Siphon):

bundle exec rails generate gitlab:click_house:siphon packages_packages --with-traversal-path
bundle exec rails generate gitlab:click_house:siphon packages_build_infos

Each invocation creates two files:

  • db/click_house/migrate/main/<TIMESTAMP>_create_siphon_<table>.rb — ClickHouse migration with the ReplacingMergeTree table definition.
  • db/siphon/tables/<table>.yml — Siphon configuration metadata file that registers the table for CDC replication.

packages_packages uses --with-traversal-path because project_id is its sharding key and we want hierarchy-aware lookups (e.g., "all packages under gitlab-org/") via the namespace_traversal_paths_dict dictionary. packages_build_infos is a join table — no traversal_path needed for v1.

Add a refresh_on_change callback so that inserts into siphon_packages_build_infos re-trigger the siphon_packages_packages row, keeping the BUILT_BY denormalization (if used) in sync. Configured in the Siphon consumer YAML, per the Siphon docs.

External repo deliverables in gitlab-org/orbit/knowledge-graph:

  1. Define a new packages domain and the Package node type in the proto schema.
  2. Define HAS_PACKAGE and BUILT_BY edge types.
  3. Wire gkg-indexer ingestion to read siphon_packages_packages and siphon_packages_build_infos from ClickHouse (apply query-time deduplication: argMax(col, _siphon_replicated_at) + filter _siphon_deleted = false, per Siphon docs).
  4. Surface counts via GET /api/v4/orbit/graph_status.

No Rails-side changes needed in the GitLab monolith — the Orbit schema is consumed via gRPC (ee/lib/analytics/knowledge_graph/grpc_client.rb) and re-exposed through ee/lib/api/orbit/data.rb. Both pick up new entities automatically once gkg-indexer advertises them. The whole Orbit surface is already gated by OrbitLicense, so no feature-flag work.

Phase 2 — File granularity + per-artifact provenance

Target LOE: S–M (~1 week)

Add PackageFile + HAS_FILE + file-level BUILT_BY. Required for SBOM-style queries like "which artifact specifically came out of pipeline X?"

bundle exec rails generate gitlab:click_house:siphon packages_package_files --with-traversal-path
bundle exec rails generate gitlab:click_house:siphon packages_package_file_build_infos

Phase 3 — Declared-dependency edge

Target LOE: S (days)

Add DECLARES_DEPENDENCY. Populated today by npm, maven, nuget, composer. Useful but distinct from CONSUMED_BY.

bundle exec rails generate gitlab:click_house:siphon packages_dependency_links
bundle exec rails generate gitlab:click_house:siphon packages_dependencies

Phase 4 — AFFECTED_BY (Package → Vulnerability)

Feasible once Package exists, because Vulnerability is already a node. Edge source: security_findings with location.dependency matching (package_name, package_type). Cross-domain, so it needs sign-off from the security domain owner.

Deferred — CONSUMED_BY (blocked on new ingestion)

The highest-value edge for AR blast-radius queries, but no current table captures (consuming_project, package) tuples. Packages::Event counts pulls for usage metrics but doesn't persist the consumer's project_id. Options:

  • New package_consumptions table populated by registry pull handlers, then CDC'd via Siphon.
  • Mine audit_events with a richer payload on registry pulls.
  • Parse lockfiles from source_code indexing (already in the graph) — partial coverage, format-dependent.

Recommend filing this as a separate issue dependent on the Package Registry team's bandwidth.

Verification

  1. gdk reconfigure && gdk restart siphon to pick up the new tables.
  2. ClickHouse::Client.select('SELECT count(*) FROM siphon_packages_packages', :main) from rails console — confirm snapshot.
  3. GET /api/v4/orbit/schema returns a packages domain with the new node and edge types.
  4. POST /api/v4/orbit/query returns Package nodes for gitlab-org/gitlab.
  5. GET /api/v4/orbit/graph_status reports a non-zero Package count for an indexed namespace.

Questions for the Orbit team

  • Are any packages_* tables already in Siphon's CDC scope? (Search of db/siphon/tables/ suggests no — only knowledge_graph_enabled_namespaces and p_knowledge_graph_code_indexing_tasks are KG-related today.)
  • Confirm Phase 1 LOE estimate (S): ~2 Siphon generator invocations + proto/schema edits in gitlab-org/orbit/knowledge-graph + indexer wiring.
  • Edge naming: should BUILT_BY be PRODUCED (Pipeline → Package) instead, to match the forward-verb pattern of existing Orbit edges (HAS_PIPELINE, BELONGS_TO)?

References

Resolved during planning (thanks @ahegyi)

  • Siphon is in production already. The public docs claiming "GDK-only" are out of date and will be updated. CDC for packages_* can ship to prod ClickHouse as soon as the YAMLs are merged.
  • NULL-engine + MV denormalization is orthogonal to Orbit. That pattern is for ClickHouse-direct analytical features. Orbit always uses graph-native nodes/edges, so BUILT_BY will be modeled as a true edge between Package and Pipeline.
Edited by 🤖 GitLab Bot 🤖