Skip to content

Add support for Yarn v2 projects

Nick Ilieskou requested to merge 263358-add-support-for-yarn-v2 into master

What does this MR do?

This MR introduces support for Yarn v2 lock files and possibly also for Yarn v3 lock files. For now we will be focusing on v2, although Yarn v3 lock files look identical to Yarn v2 lock files.

Note:

  • Every lock file definition (package) can contain a dependency, optionalDependencies and peerDependencies. For now we will not process peerDependencies. Explanation can be found on this post
  • We have added unit tests in this MR.
  • Definitions in the yarn.lock file look like this: <PACKAGE_NAME>@npm:<VERSION_REQUIREMENT>. The only packages I have seen that do not refer to npm are local workspace packages. In the code we remove the npm part of the package definition to map with the dependencies list. For example content-disposition package definition will be parsed as a Spec{Name: "content-disposition", Requirement: "0.5.4"} instead of Spec{Name: "content-disposition", Requirement: "npm:0.5.4"} which will map with the dependency of express which will look exactly the same.
"content-disposition@npm:0.5.4":
  version: 0.5.4
  resolution: "content-disposition@npm:0.5.4"
  dependencies:
    safe-buffer: 5.2.1
  checksum: 44169dbbfd5f1d48cc0cf27b1196b0da485c7d2d4b22c3c20a75babbbdd2bc6028b7afa66a9f2e2a4b14cab8506823cca227cc63e7297e17b9e9b1c00593f6db
  languageName: node
  linkType: hard

"express@npm:^4.18.2":
  version: 4.18.2
  resolution: "express@npm:4.18.2"
  dependencies:
    accepts: ~1.3.8
    array-flatten: 1.1.1
    body-parser: 1.20.1
    content-disposition: 0.5.4
......
  • An entry with all the fields present look like the following:
"<PACKAGE_NAME>@npm:<VERSION_REQUIREMENT>":     -------> We parse this
  version: <VERSION                             -------> We parse this
  resolution: ...
  dependencies:                                 -------> We parse this
    ....
    ....
  dependenciesMeta:
    ....
      ...
  peerDependencies:
    ...                                         -------> This is turned off for now
  peerDependenciesMeta:
    ...
  bin:
    ...
  linkType: <hard || soft> 
  checksum: ....
  languageName: ....
  • We parse exactly the same information as we do for Yarn v1. So all data structures are exactly the same.
  • We ignore workspace related packages. We will introduce workspace packages with gitlab-org/gitlab#394822
  • Remediation for yarn v2 lock files is skipped. We will introduce this feature at a later point. Related issue: gitlab-org/gitlab#395011
  • Tested:
    • Tested against js-yarn to ensure that Yarn v1 still works. Note that we need to give a bigger remediation timeout for DS to succeed.
    • Tested against js-yarn-v2 for the new changes introduced by this MR.

Files that were modified:

  • remediate/yarn.go : The only change here is that we want to support remediation only for Yarn classic. For that reason we check the yarn flavor before doing remediation
  • qa/: Added expect and fixtures for testing Berry Yarn v2. No tests have been added for Yarn v3. If you want we can add them in a separate MR.
  • rspec/ : Integration test added for Berry Yarn v2
  • scanner/parser/yarn:
    • expect and fixutres have been updated with a new strucutre based on berry and classic. These are being used by unit tests in yarn_test.go
    • yarn.go: In this file we figure out what yarn flavor we have and we call the respective file to process the lock file. We call berry.go for berry flavor and classic.go for classic flavor. Notice that in the Parse method we need to create to buffers with the contents of the file since we are using io.Reader. We need to read the interface once to identify which yarn flavor we have and once more for processing the lock file. io.Reader cannot be reverted and that is why we store the content in the buffers.
    • classic.go: Most of the code of the old yarn.go is now here. Nothing is changed. Unit tests supports almost 100% coverage.
    • berry.go: For berry flavor we parse the yaml file and then for each YamlPackage we make sure that the name is parsed correctly
  • Added unit tests for yarn v3.

What are the relevant issue numbers?

gitlab-org/gitlab#263358 (closed), gitlab-org/gitlab#351841 (closed)

Does this MR meet the acceptance criteria?

Edited by Nick Ilieskou

Merge request reports