Temporarily Exclude deps.dev Source from Processor Handling
Problem
Our License DB system currently processes data indiscriminately from all sources. As we integrate data from deps.dev
, it is necessary to modify the Processor to temporarily ignore this new source. This ensures that we manage data integration and testing in controlled phases.
Solution
Adjust the Processor to exclude licenses sourced from deps.dev
while continuing to handle data from repo
and the default source. This selective processing allows for phased testing and integration without disrupting the existing data flow.
Implementation Plan
-
Modify the License
struct to include aSource
field to identify the data source. -
Update the Server
handling logic to bypass processing for licenses with a source labeled asdeps.dev
. -
Implement logging for bypassed deps.dev
licenses to monitor and verify that they are being correctly identified and excluded.
Outline
Here is a guide indicating the essential code changes:
diff --git a/data/license.go b/data/license.go
index 8c92e3c..0797f1c 100644
--- a/data/license.go
+++ b/data/license.go
@@ -37,6 +37,7 @@ type License struct {
APIVersion string `json:"api_version"`
PackageRegistry string `json:"package_registry"`
Versions []*Version `json:"versions"`
+ Source string `json:"source"` // repo or deps.dev
}
// NewLicense creates a new license for the specified packageRegistry
diff --git a/server.go b/server.go
index 39841b1..6d328ef 100644
--- a/server.go
+++ b/server.go
@@ -63,6 +63,12 @@ func (s *Server) Incoming(w http.ResponseWriter, r *http.Request) {
Int("version_count", len(license.Versions)).
Logger()
+ if license.Source == "deps.dev" {
+ requestLogger.Warn().Msg("deps.dev license received but not processed")
+ w.WriteHeader(http.StatusOK)
+ return
+ }
+
if len(license.Versions) == 0 {
requestLogger.Warn().Msg("no version data found")
w.WriteHeader(http.StatusOK)
Edited by Philip Cunningham