Engineering research - Decide next language to add to Dependency Scanning

added Category:Dependency Scanning [DEPRECATED] devopssecure groupcomposition analysis labels

changed milestone to %13.1

@gonzoyumo as this is engineering research should we time box it? I'd like engineering research done 13.1 and the chosen language researched and POC'd in 13.2

Timeboxing to ~1 week at max.

@fcatteau I'd like to take this opportunity to write down a list of requirements to add support for a new language/package manager. This would allow anyone to quickly identify if adding support is doable and possibly having it handled by the wider community. Thanks!

I'd like to take this opportunity to write down a list of requirements to add support for a new language/package manager.

@gonzoyumo Great! I've added this as a task for this issue. See proposal.

added workflowdesign label

changed the description

added [deprecated] Accepting merge requests label

added AST Leadership label

added backend label

changed milestone to %13.2

changed milestone to %13.1

changed title from Decide next language to add to Dependency Scanning to Engineering research - Decide next language to add to Dependency Scanning

set weight to 5

assigned to @fcatteau

removed [deprecated] Accepting merge requests label

changed the description

Requirements

I'm taking some notes on the requirements, and this is a draft for what should eventually land in the handbook. cc @gonzoyumo

Gemnasium integration

To be integrated into Gemnasium, a package manager must be able to generate a lock file that lists all the project dependencies, including the transient dependencies (dependencies of dependencies explicitly requested by the user). The dependency list must contain the package names and versions, so that the dependencies can be compared to the advisories of gemnasium-db, the Gemnasium Vulnerability Database. Dependencies must have comparable versions, so that Gemnasium can establish if the version the project depends on matches the range of affected versions, in the vulnerability database.

If the projects supported by the package manager usually have no lock file, there must be a command line that generates a lock file for any given dependency file. If this command fetches packages from external registries, then the command line must have settings for the proxies to these registries, and for custom certificates to use when connecting to these over HTTPS. The proxies and custom certificates can either be given as CLI options or environment variables (preferred), or be set in a configuration file.

Scanner integration

To be integrated into GitLab Dependency Scanning, a security scanner supporting a package manager must be able to generate all the mandatory fields defined in the Dependency Scanning JSON schema. There should be no technical constraint preventing the scanner from being bundled in a Docker image. It should be possible to build the Docker image using a Dockerfile, at any time and without any human intervention. If the scanner fetches external resources or connects to remote servers during its execution, it must provide a way to define proxies for these resources and servers, and to use custom certificates. See Secure Scanner integration guide.

The requirements for Scanner integration should probably be integrated into the Secure Scanner integration guide. Actually, what I've written in my previous comment is totally generic, except for a reference to the Dependency Scanning JSON schema, which the integration guide should already have.

The requirements for Gemnasium integration should probably part of https://docs.gitlab.com/ee/development/integrations/.

Also, the Secure Scanner integration guide should be updated with requirements for offline mode support. I don't know whether offline mode support is required though.

I've created two checklists I'm going to use when evaluating package managers for .NET, C, and C++. cc @xlgmokha

Gemnasium integration checklist

The package managers meets at least one of these two conditions:

Supported projects have a lock file that lists transient and non-transient dependencies.
There is a CLI command that generates such a lock file, and this CLI and its dependencies can be bundled in a Linux-based Docker image.

requirements for lock files:

predictable file names
contain both transient and direct dependencies
give dependency name
give dependency version
provide a format version, or another way of telling if the file format is supported
give dependencies of dependencies, in order to build a dependency graph (optional, see #198034 (closed))

requirements for package versions:

package versions are comparable (three-way comparison can be implemented)

requirements for package names:

package names can be compared; false-positives are acceptable
package names can be resolved to package URLs, or package URLs are provided

requirements for vulnerability database:

When a security advisory is published on NVD, it's possible to tell this relates to a package handled by the package manager. The name of the vulnerable package (as listed in the lock file) can be extracted.

Dependency Scanning Tool integration checklist

can run as a CLI (no GUI)
can run in a Linux-based Docker image
generates all the required fields defined in the Dependency Scanning JSON schema
can run offline, or using custom proxies and custom certificates
only has allowed licenses

Dependency Scanning Tool integration checklist

Actually, we already have this checklist in the handbook, and we're currently updating it. See gitlab-com/www-gitlab-com!52190 (merged)

I've updated the Gemnasium integration checklist, to make sure package names can be compared to security advisories, in the Gemnasium Vulnerability DB. cc @julianthome

changed the description

mentioned in merge request gitlab-com/www-gitlab-com!52190 (merged)

added quad-planningcomplete-no-action label

NuGet

See #8102 (comment 281626866) for NuGet support. cc @xlgmokha

I suggest we start with packages.lock.json support, even though generation of these lock files has to be enabled, and even though this feature is available with NuGet 4.9 or above and with Visual Studio 2017 15.9 or above. Hopefully the documentation can make explicitly that NuGet is supported only via packages.lock.json, until we implement NuGet project file support.

I was not able to find the JSON schema for packages.lock.json lock files.

I've found a sample .NET Core 2.2 project, with a .csproj file and a packages.lock.json file. Here's an excerpt:

{
  "version": 1,
  "dependencies": {
      "Microsoft.NETCore.DotNetHostPolicy": {
        "type": "Transitive",
        "resolved": "2.2.0",
        "contentHash": "5Xvh/3bDr6YAl6TKfeONvRDZ9QOmvVmzFzA0M6g8uubWdf5/o6qdVOqByFBT/fhjVb6okP0E5+v1oxh1Pk+c+w==",
        "dependencies": {
          "Microsoft.NETCore.DotNetHostResolver": "2.2.0"
        }
      },
      "Microsoft.NETCore.DotNetHostResolver": {
        "type": "Transitive",
        "resolved": "2.2.0",
        "contentHash": "Q3j3KC2ugqIVasf7pO4NRDEn7GysZX3ZH6fAHfbjrP8cYXY9cmHeFcbaniw36q8kFhsPt2EnRHzSsLBpbG6l2Q==",
        "dependencies": {
          "Microsoft.NETCore.DotNetAppHost": "2.2.0"
        }
      }
}

As we can see, the dependencies (top-level key) is a JSON object where the keys are the package names, and the values are JSON objects that give the resolved version and a list of dependencies. We have the information needed to implement packages.lock.json support in Gemnasium. cc @xlgmokha

Also, the version top-level key makes possible to check whether the file format is supported.

Also, the version top-level key makes possible to check whether the file format is supported.

This is excellent @fcatteau. I think this is a good starting point.

FYI: With this information it's also possible to fetch license information from "https://api.nuget.org/v3-flatcontainer/#{name}/#{version}/#{name}.nuspec". That can come in a future iteration but I thought I would share in case you were interested @fcatteau.

@fcatteau Will the name of the file always be packages.lock.json or is that controlled through configuration? Is there a single lock file for a solution or will there be one for each project attached to a solution.

For example:

* myapp.sln
  * /src/
    * web/
      * web.csproj
      * packages.lock.json
    * service/
      * service.csproj
      * packages.lock.json
    * domain/
      * domain.csproj
      * packages.lock.json

or...

* myapp.sln
* packages.lock.json
  * /src/
    * web/
      * web.csproj
    * service/
      * service.csproj
    * domain/
      * domain.csproj

I'm asking because I was wondering if we need to do a recursive scan into project directories or if we can expect the file to be in the root next to the solution file. Also, I'm not sure if solution files are still used or not. dotnet cli does seem to offer generating one, so I assume yes, but I would like to suggest that we double check.

@xlgmokha AFAIK the lock file lives with the project file, and not with the solution file. So there might be multiple lock files in a repo, which is fine since gemnasium already looks for lock files recursively. To be double checked though.

In any case, I see no reason for parsing the solution file, and hopefully this reduces the complexity of NuGet support.

FYI: With this information it's also possible to fetch license information from "https://api.nuget.org/v3-flatcontainer/#{name}/#{version}/#{name}.nuspec". That can come in a future iteration but I thought I would share in case you were interested

Yes, we might need to connect to the package registry in order to track new releases, and Show when a component is out of date. It's out of scope but it's definitely worth sharing - thank you!

AFAIK the lock file lives with the project file, and not with the solution file.

@fcatteau I assume the same. I just wanted to make sure.

In any case, I see no reason for parsing the solution file, and hopefully this reduces the complexity of NuGet support

Yep, good point. The solution file points to the project files but if we're recursively scanning for the project files then there is no need to parse the solution file.

@julianthome Do you foresee any complication when adding security advisories for NuGet packages to gemnasium-db?

Awesome! Having NuGet support would be great. I think there are some related issues regarding NuGet support:

Quoting from #8102 (comment 325267618):

We recently introduced issue templates on gemnasium-db for Package Type Support Request and Schema Change Request issues so that we have a place where we can sort out the details of information that is required by the analyzer upfront. I took the liberty of creating one related to NuGet for groupcomposition analysis .

Depending on when this gets planned, we would be grateful if someone from the Composition Analysis Team could fill out the package type support request for nuget and provide us some details about the required information from the analyzers' perspective. Common questions and the corresponding explanations are part of the package type support request template.

NuGet uses Semantic Versioning 2.0.0, and a Maven-like syntax for version ranges. All that could be supported in the Gemnasium vrange library since we already have SemVer support and Maven support. We might have to implement a new vrange plugin that combines SemVer versions with Maven version ranges though. We might be able to implement that in Go, in gemnasium/semver. cc @julianthome

Conan

Conan provides a conan graph commands that generates a lock file, usually named conan.lock. See https://docs.conan.io/en/latest/reference/commands/misc/graph.html

I couldn't find the schema for conan.lock files though.

The path of a Conan dependency combines a package name, a version, a user, and a channel. It's something like mypkg/0.1@user/channel. See requires attribute.

@julianthome I suggest we ignore the user and the channel when matching the package with a security advisory of the Gemnasium Vulnerability DB. We might have false-positives, but it's better than having false-negatives. I still have to learn more on Conan channels though.

There doesn't seem to be a central Conan registry, but https://github.com/bincrafters seems a major registry.

Actually it looks like Conan has a central repo for OSS libraries. See this introduction to Conan by Bincrafters:

Conan has the following important characteristics:

A moderated and trusted central repository for OSS software

I believe they're referring to Bincrafters itself.

The central repo for Conan packages is conan.io, and the corresponding Conan channel is conan. See opencv/4.1.1@conan/stable for instance. Interestingly, the Conan user is used to indicate the stability. cc @julianthome

I'll watch these three-part presentation of Conan later on this week, if time permits: https://www.youtube.com/watch?v=xBLjXdyh3zs

Here's a conan.lock I was able to generate for a minimalistic conanfile.txt.

{
  "profile_host": "[settings]\narch=x86_64\narch_build=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++\ncompiler.version=6.1\nos=Linux\nos_build=Linux\n[options]\n[build_requires]\n[env]\n",
  "graph_lock": {
    "nodes": {
      "0": {
        "pref": null,
        "options": "openssl:386=False\nopenssl:fPIC=True\nopenssl:no_asm=False\nopenssl:no_async=False\nopenssl:no_bf=False\nopenssl:no_cast=False\nopenssl:no_des=False\nopenssl:no_dh=False\nopenssl:no_dsa=False\nopenssl:no_dso=False\nopenssl:no_hmac=False\nopenssl:no_md2=False\nopenssl:no_md5=False\nopenssl:no_mdc2=False\nopenssl:no_rc2=False\nopenssl:no_rc4=False\nopenssl:no_rc5=False\nopenssl:no_rsa=False\nopenssl:no_sha=False\nopenssl:no_sse2=False\nopenssl:no_threads=False\nopenssl:no_zlib=False\nopenssl:openssldir=None\nopenssl:shared=False\npoco:cxx_14=False\npoco:enable_apacheconnector=False\npoco:enable_cppparser=False\npoco:enable_crypto=True\npoco:enable_data=True\npoco:enable_data_mysql=False\npoco:enable_data_odbc=False\npoco:enable_data_sqlite=True\npoco:enable_encodings=True\npoco:enable_json=True\npoco:enable_mongodb=True\npoco:enable_net=True\npoco:enable_netssl=True\npoco:enable_netssl_win=True\npoco:enable_pagecompiler=False\npoco:enable_pagecompiler_file2page=False\npoco:enable_pdf=False\npoco:enable_pocodoc=False\npoco:enable_redis=True\npoco:enable_sevenzip=False\npoco:enable_tests=False\npoco:enable_util=True\npoco:enable_xml=True\npoco:enable_zip=True\npoco:fPIC=True\npoco:force_openssl=True\npoco:poco_unbundled=False\npoco:shared=False\nzlib:fPIC=True\nzlib:minizip=False\nzlib:shared=False",
        "requires": [
          "1",
          "2"
        ],
        "path": "/mypkg2/conanfile.txt"
      },
      "1": {
        "pref": "poco/1.9.4#0:14552302cafb13b446cac0107fd15a20ecde247b",
        "options": "cxx_14=False\nenable_apacheconnector=False\nenable_cppparser=False\nenable_crypto=True\nenable_data=True\nenable_data_mysql=False\nenable_data_odbc=False\nenable_data_sqlite=True\nenable_encodings=True\nenable_json=True\nenable_mongodb=True\nenable_net=True\nenable_netssl=True\nenable_netssl_win=True\nenable_pagecompiler=False\nenable_pagecompiler_file2page=False\nenable_pdf=False\nenable_pocodoc=False\nenable_redis=True\nenable_sevenzip=False\nenable_tests=False\nenable_util=True\nenable_xml=True\nenable_zip=True\nfPIC=True\nforce_openssl=True\npoco_unbundled=False\nshared=False\nopenssl:386=False\nopenssl:fPIC=True\nopenssl:no_asm=False\nopenssl:no_async=False\nopenssl:no_bf=False\nopenssl:no_cast=False\nopenssl:no_des=False\nopenssl:no_dh=False\nopenssl:no_dsa=False\nopenssl:no_dso=False\nopenssl:no_hmac=False\nopenssl:no_md2=False\nopenssl:no_md5=False\nopenssl:no_mdc2=False\nopenssl:no_rc2=False\nopenssl:no_rc4=False\nopenssl:no_rc5=False\nopenssl:no_rsa=False\nopenssl:no_sha=False\nopenssl:no_sse2=False\nopenssl:no_threads=False\nopenssl:no_zlib=False\nopenssl:openssldir=None\nopenssl:shared=False\nzlib:fPIC=True\nzlib:minizip=False\nzlib:shared=False",
        "requires": [
          "2"
        ]
      },
      "2": {
        "pref": "openssl/1.0.2u#0:61859e8300e1dcdc224de5a8ea12bf514c17762c",
        "options": "386=False\nfPIC=True\nno_asm=False\nno_async=False\nno_bf=False\nno_cast=False\nno_des=False\nno_dh=False\nno_dsa=False\nno_dso=False\nno_hmac=False\nno_md2=False\nno_md5=False\nno_mdc2=False\nno_rc2=False\nno_rc4=False\nno_rc5=False\nno_rsa=False\nno_sha=False\nno_sse2=False\nno_threads=False\nno_zlib=False\nopenssldir=None\nshared=False\nzlib:fPIC=True\nzlib:minizip=False\nzlib:shared=False",
        "requires": [
          "3"
        ]
      },
      "3": {
        "pref": "zlib/1.2.11#0:e91aebf002c7f10410b89969864731d7adbad3ad",
        "options": "fPIC=True\nminizip=False\nshared=False"
      }
    }
  },
  "version": "0.3"
}

My guess is that libz is a transient dependency, even though it's not explicitly listed in any of the recipes. See https://conan.io/center/poco/1.9.4/ and https://conan.io/center/openssl/1.0.2u/

Also, each node comes with its list of dependencies (what it requires), and the first node represents the project itself. All that makes it easy to build a dependency graph. See #198034 (closed).

The image name and version can easily be extracted from the pref field.

Here's corresponding the conanfile.txt:

[requires]
poco/1.9.4
openssl/1.0.2u

I got the lock file by running this command after installing Conan:

conan graph lock -s compiler=gcc -s compiler.libcxx=libstdc++ -s compiler.version=6.1 conanfile.txt

I installed Conan by running pip install conan in a Docker container, using Docker image python:3-slim.

cc @xlgmokha

@julianthome Do you foresee any complication when adding security advisories for Conan packages to gemnasium-db?

No not really. We would be very grateful though if you could create a corresponding Package Type Request issue on gemnasium-db (by selecting the Package Type Request issue template) which would help us to understand the type of data you need for adding conan support.

@julianthome Great! I'll open a Package Type Request issue on gemnasium-db then. TODO @fcatteau

Awesome! Thanks a lot @fcatteau

I didn't say what it would take to add Conan support to Gemnasium's vrange library. This should be straightforward since Conan uses node-semver (Python package) in its version ranges. Actually, we should be able to reuse vrange/npm without any modification. cc @julianthome

By the way, in order to implement Conan support for Dependency Scanning we would end up adding security advisories for system libraries to gemnasium-db. This would be a step toward implementing Container Scanning on top of gemnasium-db. Actually, there might be an issue about that, and I might be the author of the issue. cc @julianthome

changed health status to on track

marked the checklist item write down a list of requirements to add support for a new language/package manager as completed

changed the description

@NicoleSchwartz I've updated the issue with the discovery conclusion. I'm not sure users usually keep lock files in the repos of their NuGet or Conan projects, or if they would be willing to do that, but in any case lock file support is the first step before supporting the main build/dependency file. (Also keeping lock files in the repos is considered as a best practice.) cc @gonzoyumo

marked this issue as related to #201781 (closed)

marked this issue as related to #8102 (closed)

changed the description

@NicoleSchwartz In the end I considered we didn't need a PoC to check feasibility. We know enough about the lock files Conan and NuGet generate to prove that these can be supported. I've updated the proposal accordingly. cc @gonzoyumo

@fcatteau if I'm reading correctly the conclusion, there is no preference from a technical point of view to start with .Net Framework/Core/C# (with NuGet) or C/C++ (with Conan), right?

I'm closing this research as work is done. @NicoleSchwartz let's catch up to figure out which one to start with and organize epics accordingly.

@gonzoyumo You're right. Both NuGet and Conan can be supported, at a comparable cost. I'll add that to the conclusions of this discovery. cc @NicoleSchwartz

closed

mentioned in issue #217023 (closed)

changed the description

mentioned in issue #225218 (closed)

mentioned in issue #225219 (closed)

mentioned in issue #207325 (closed)

mentioned in issue #8102 (closed)

mentioned in issue #217374 (closed)

removed the relation with #201781 (closed)

removed the relation with #8102 (closed)

changed the description

added Enterprise Edition GitLab Ultimate labels

added Category:Software Composition Analysis SCA:Dependency Scanning labels

Engineering research - Decide next language to add to Dependency Scanning

Problem to solve

Intended users

User experience goal

Proposal

Discovery conclusion

Links / references

Designs

Child items ...

Activity

Requirements

Gemnasium integration

Scanner integration

Gemnasium integration checklist

Dependency Scanning Tool integration checklist

NuGet

Conan