Orbit: Add Package Registry entities and edges to the knowledge graph
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Context
The Orbit knowledge graph currently has zero artifact entities. The server-side schema covers ci, code_review, core, plan, security, and source_code domains, but nothing for packages, container images, or deployments.
During the AR agentic spike (#600994) we found that answering "what's the blast radius of @gitlab/ui?" required 20+ REST bridge calls. A single Orbit traversal would suffice if Package existed as a node with a BUILT_BY edge to Pipeline.
Package Registry data lives in the GitLab monolith database (packages_packages, packages_package_files, packages_build_infos, etc.) and is replicated to ClickHouse via Siphon. gkg-indexer is the downstream consumer that materializes the Orbit graph from those ClickHouse tables.
Proposed entities
Package— sourced frompackages_packages- Properties:
name,version,package_type(npm, maven, pypi, nuget, golang, cargo, generic, debian, conan, rubygems, helm, terraform_module, rpm, composer, ml_model),status,project_id,created_at,updated_at
- Properties:
PackageFile— sourced frompackages_package_files- Properties:
file_name,size,file_sha256,status,created_at
- Properties:
Proposed edges
HAS_PACKAGE—Project→Package- Source:
packages_packages.project_id - Impact: basic package-to-project ownership
- Source:
HAS_FILE—Package→PackageFile- Source:
packages_package_files.package_id - Impact: package contents
- Source:
BUILT_BY(package-level) —Package→Pipeline- Source:
packages_build_infos(package_id, pipeline_id)join table - Impact: package provenance ("which pipeline built this artifact")
- Source:
BUILT_BY(file-level) —PackageFile→Pipeline- Source:
packages_package_file_build_infos(package_file_id, pipeline_id)join table - Impact: per-artifact provenance for SBOM-style queries
- Source:
DECLARES_DEPENDENCY—Package→(name, version_pattern)- Source:
packages_dependency_links+packages_dependencies - Impact: declared external deps; populated today by npm, maven, nuget, composer
- Source:
CONSUMED_BY—Package→Project- Source: not derivable from current tables — requires new ingestion (see Deferred work)
- Impact: highest-impact edge for AR. Enables blast-radius queries in a single traversal
AFFECTED_BY—Package→Vulnerability- Source:
security_findingswithlocation.dependencymatched on(package_name, package_type) - Impact: package-to-vulnerability linkage
- Source:
Plan
Phase 1 — Minimum viable: provenance edge
Target LOE: S (days–1 week)
Ship Package + HAS_PACKAGE + BUILT_BY (package-level). This is the smallest slice that lets agents answer "which pipeline built this package?" — the spike's primary unmet question. Package-level BUILT_BY (via packages_build_infos) is preferred over file-level for v1 because it's a single, narrower edge with smaller fan-out per package.
Monolith deliverables — generate Siphon tables via the documented Rake task (ClickHouse table design with Siphon):
bundle exec rails generate gitlab:click_house:siphon packages_packages --with-traversal-path
bundle exec rails generate gitlab:click_house:siphon packages_build_infosEach invocation creates two files:
db/click_house/migrate/main/<TIMESTAMP>_create_siphon_<table>.rb— ClickHouse migration with theReplacingMergeTreetable definition.db/siphon/tables/<table>.yml— Siphon configuration metadata file that registers the table for CDC replication.
packages_packages uses --with-traversal-path because project_id is its sharding key and we want hierarchy-aware lookups (e.g., "all packages under gitlab-org/") via the namespace_traversal_paths_dict dictionary. packages_build_infos is a join table — no traversal_path needed for v1.
Add a refresh_on_change callback so that inserts into siphon_packages_build_infos re-trigger the siphon_packages_packages row, keeping the BUILT_BY denormalization (if used) in sync. Configured in the Siphon consumer YAML, per the Siphon docs.
External repo deliverables in gitlab-org/orbit/knowledge-graph:
- Define a new
packagesdomain and thePackagenode type in the proto schema. - Define
HAS_PACKAGEandBUILT_BYedge types. - Wire
gkg-indexeringestion to readsiphon_packages_packagesandsiphon_packages_build_infosfrom ClickHouse (apply query-time deduplication:argMax(col, _siphon_replicated_at)+ filter_siphon_deleted = false, per Siphon docs). - Surface counts via
GET /api/v4/orbit/graph_status.
No Rails-side changes needed in the GitLab monolith — the Orbit schema is consumed via gRPC (ee/lib/analytics/knowledge_graph/grpc_client.rb) and re-exposed through ee/lib/api/orbit/data.rb. Both pick up new entities automatically once gkg-indexer advertises them. The whole Orbit surface is already gated by OrbitLicense, so no feature-flag work.
Phase 2 — File granularity + per-artifact provenance
Target LOE: S–M (~1 week)
Add PackageFile + HAS_FILE + file-level BUILT_BY. Required for SBOM-style queries like "which artifact specifically came out of pipeline X?"
bundle exec rails generate gitlab:click_house:siphon packages_package_files --with-traversal-path
bundle exec rails generate gitlab:click_house:siphon packages_package_file_build_infosPhase 3 — Declared-dependency edge
Target LOE: S (days)
Add DECLARES_DEPENDENCY. Populated today by npm, maven, nuget, composer. Useful but distinct from CONSUMED_BY.
bundle exec rails generate gitlab:click_house:siphon packages_dependency_links
bundle exec rails generate gitlab:click_house:siphon packages_dependenciesPhase 4 — AFFECTED_BY (Package → Vulnerability)
Feasible once Package exists, because Vulnerability is already a node. Edge source: security_findings with location.dependency matching (package_name, package_type). Cross-domain, so it needs sign-off from the security domain owner.
Deferred — CONSUMED_BY (blocked on new ingestion)
The highest-value edge for AR blast-radius queries, but no current table captures (consuming_project, package) tuples. Packages::Event counts pulls for usage metrics but doesn't persist the consumer's project_id. Options:
- New
package_consumptionstable populated by registry pull handlers, then CDC'd via Siphon. - Mine
audit_eventswith a richer payload on registry pulls. - Parse lockfiles from
source_codeindexing (already in the graph) — partial coverage, format-dependent.
Recommend filing this as a separate issue dependent on the Package Registry team's bandwidth.
Verification
gdk reconfigure && gdk restart siphonto pick up the new tables.ClickHouse::Client.select('SELECT count(*) FROM siphon_packages_packages', :main)from rails console — confirm snapshot.GET /api/v4/orbit/schemareturns apackagesdomain with the new node and edge types.POST /api/v4/orbit/queryreturnsPackagenodes forgitlab-org/gitlab.GET /api/v4/orbit/graph_statusreports a non-zeroPackagecount for an indexed namespace.
Questions for the Orbit team
- Are any
packages_*tables already in Siphon's CDC scope? (Search ofdb/siphon/tables/suggests no — onlyknowledge_graph_enabled_namespacesandp_knowledge_graph_code_indexing_tasksare KG-related today.) - Confirm Phase 1 LOE estimate (S): ~2 Siphon generator invocations + proto/schema edits in
gitlab-org/orbit/knowledge-graph+ indexer wiring. - Edge naming: should
BUILT_BYbePRODUCED(Pipeline → Package) instead, to match the forward-verb pattern of existing Orbit edges (HAS_PIPELINE,BELONGS_TO)?
References
- ClickHouse table design with Siphon
- Spike findings: #600994
- Knowledge graph spike: #600729 (closed)
- Agentic vision: &21864
Resolved during planning (thanks @ahegyi)
- Siphon is in production already. The public docs claiming "GDK-only" are out of date and will be updated. CDC for
packages_*can ship to prod ClickHouse as soon as the YAMLs are merged. - NULL-engine + MV denormalization is orthogonal to Orbit. That pattern is for ClickHouse-direct analytical features. Orbit always uses graph-native nodes/edges, so
BUILT_BYwill be modeled as a true edge betweenPackageandPipeline.