You need to sign in or sign up before continuing.
Create scheduled pipelines to run cargo feeder and exporter
Problem to solve
To begin ingestion of a new package registry type, the license-feeder must be triggered via a scheduled pipeline. And to export the new data, the exporter should be triggered in a similar way.
Proposal
Add Scheduled Pipelines for cargo
license-feeder and license-exporter.
Implementation plan
Follow process documented in Creating scheduled pipelines for license-feeder to create a scheduled pipeline for a cargo
feeder and exporter, both for dev
and prod
.
-
Update list of registries in run_exporter.sh
-
Update pipelines
-
Run feeder with
ignore_cursor: all
Verification steps
- Data is in the database after feeder has been run.
- Data is exporter to the license bucket after exporter has been run.
- Cardinality of
cargo_component
is approximately the same as that of uniqueName
forSystem=CARGO
under the PackageVersions table.
Edited by Igor Frenkel