Extracting mappings (package slug -> source URL) from Gemnasium Postgres DB

Problem to solve

Due to the elimination of the Gemnasium Client/Server architecture, analyzers leverage the gemnasium-db repo directly instead of connecting to the Gemnasium API. Due to this change, the Gemnasium Postgres DB will become obsolete in the near future.

However, some information contained in the Gemnasium Postgres DB database is still valuable. One piece of information that would be useful with respect to advisory generation is the mapping between package-slugs and their source location/URL (e.g., on GitHub).

Intended users

• Sam (Security Analyst)

Proposal

For the use-case of extracting a mapping from package-slugs to their source location, a SQL dump from Gemnasium Postgres DB that contains this information would be sufficient. For solving this problem, we do not need to keep the Gemnasium architecture (including the package sync jobs) alive as we would not gain any new information by doing so: the package registries are lazily mirrored (based on package usage); as the Gemnasium-API won't be used anymore, there won't be any newly added packages and, thus, no newly learned mappings.

The purpose of the SQL dump would be to bootstrap a source URL map (similar to our CPE Map) that keeps track of the relation between packages and their source location.

Links / references

/cc @fcatteau @gonzoyumo @NicoleSchwartz @plafoucriere

added typefeature label

changed the description

I'm not sure to understand completely this issue:

For the use-case of extracting a mapping from package-slugs to their source location, a SQL dump from Gemnasium Postgres DB that contains this information would be sufficient. For solving this problem, we do not need to keep the Gemnasium architecture (including the package sync jobs) alive [...]

How do you keep your data up-to-date then? Projects URL can move with time, and you don't have any way to follow this if you don't have a service running or a regular job. How do you plan to maintain this data?

How do you keep your data up-to-date then? Projects URL can move with time, and you don't have any way to follow this if you don't have a service running or a regular job.

Yes, that is true. The extracted relation from the Gemnasium Postgres DB would be only used to bootstrap an initial map (package slug -> source URL) which we could use as a starting point. We could then implement a simpler mechanism that checks for updated (i.e., source URL changes) on a regular basis; we could still re-use part of the code from the Gemnasium Server for that. This problem is actually pretty similar to the CPE map maintenance. The CPE map stores a mapping between (vendor id -> package slugs) which we are using for advisory generation.

Extracting the source urls based on a package name is not that complicated for most of the package registries (pypi, npm, packagist, go, gem). We could realise this with a small ruby script (scheduled pipeline) and it would be sufficient to use a text file or maybe a small SQLite db to store the data.

Thanks for the details. That's what I'm afraid of: tearing down the current services today, and recreating something similar from scratch, because the scope is smaller. We can stop and archive some of the services (the web API for example, and stop exposing services to the Internet).

changed the description

added Category:GitLab Advisory Database devopssecure groupcomposition analysis labels

To me this is a ~backstage issue, not a  feature. @julianthome WDYT?

Yes, thanks @fcatteau ... going to change the label accordingly.

added backstage [DEPRECATED] label and removed typefeature label

I took the liberty of adding this particular issue to the next Secure Brainstorming Session (2019/01/14)

Unfortunately, I cannot attend the brainstorming session next Tuesday. Perhaps we can continue the discussion in this issue for the time being and schedule an extra Zoom for that. I suppose this is not a super urgent matter anyway.

Based on the discussion we already started in the issue above, there is one main question that we may have to sort-out in my opinion: would the maintenance effort that goes into keeping the Gemnasium Infrastructure alive (e.g., changing API of integrated PRs upon changes, adding new PRs, maintaining adapting db schema) justify the benefit of getting updated information regarding the source URL of a package?

The main advantage of keeping the infrastructure alive would be to get updated source information for cases where the source URL of a package changes. There is a second, related question to that: Does it happen very frequently that the source URL of a package changes?

added 1 deleted label and removed groupcomposition analysis label

@dappelt pointed me to https://libraries.io/ which provides a huge data-set containing all the data we need for extracting the source location of a given package.

Closing this issue - please feel free to reopen it when needed.

closed

added groupdynamic analysis label and removed 1 deleted label