Add support for Yarn v2 projects
What does this MR do?
This MR introduces support for Yarn v2 lock files and possibly also for Yarn v3 lock files. For now we will be focusing on v2, although Yarn v3 lock files look identical to Yarn v2 lock files.
Note:
- Every lock file definition (package) can contain a
dependency
,optionalDependencies
andpeerDependencies
. For now we will not processpeerDependencies
. Explanation can be found on this post - We have added unit tests in this MR.
- Definitions in the yarn.lock file look like this:
<PACKAGE_NAME>@npm:<VERSION_REQUIREMENT>
. The only packages I have seen that do not refer to npm are local workspace packages. In the code we remove thenpm
part of the package definition to map with the dependencies list. For example content-disposition package definition will be parsed as aSpec{Name: "content-disposition", Requirement: "0.5.4"}
instead ofSpec{Name: "content-disposition", Requirement: "npm:0.5.4"}
which will map with the dependency ofexpress
which will look exactly the same.
"content-disposition@npm:0.5.4":
version: 0.5.4
resolution: "content-disposition@npm:0.5.4"
dependencies:
safe-buffer: 5.2.1
checksum: 44169dbbfd5f1d48cc0cf27b1196b0da485c7d2d4b22c3c20a75babbbdd2bc6028b7afa66a9f2e2a4b14cab8506823cca227cc63e7297e17b9e9b1c00593f6db
languageName: node
linkType: hard
"express@npm:^4.18.2":
version: 4.18.2
resolution: "express@npm:4.18.2"
dependencies:
accepts: ~1.3.8
array-flatten: 1.1.1
body-parser: 1.20.1
content-disposition: 0.5.4
......
- An entry with all the fields present look like the following:
"<PACKAGE_NAME>@npm:<VERSION_REQUIREMENT>": -------> We parse this
version: <VERSION -------> We parse this
resolution: ...
dependencies: -------> We parse this
....
....
dependenciesMeta:
....
...
peerDependencies:
... -------> This is turned off for now
peerDependenciesMeta:
...
bin:
...
linkType: <hard || soft>
checksum: ....
languageName: ....
- We parse exactly the same information as we do for Yarn v1. So all data structures are exactly the same.
- We ignore workspace related packages. We will introduce workspace packages with gitlab-org/gitlab#394822
- Remediation for yarn v2 lock files is skipped. We will introduce this feature at a later point. Related issue: gitlab-org/gitlab#395011
-
Tested:
- Tested against js-yarn to ensure that Yarn v1 still works. Note that we need to give a bigger remediation timeout for DS to succeed.
- Tested against js-yarn-v2 for the new changes introduced by this MR.
Files that were modified:
-
remediate/yarn.go
: The only change here is that we want to support remediation only for Yarn classic. For that reason we check the yarn flavor before doing remediation -
qa/
: Added expect and fixtures for testing Berry Yarn v2. No tests have been added for Yarn v3. If you want we can add them in a separate MR. -
rspec/
: Integration test added for Berry Yarn v2 -
scanner/parser/yarn
:-
expect
andfixutres
have been updated with a new strucutre based on berry and classic. These are being used by unit tests inyarn_test.go
-
yarn.go
: In this file we figure out what yarn flavor we have and we call the respective file to process the lock file. We callberry.go
for berry flavor andclassic.go
for classic flavor. Notice that in theParse
method we need to create to buffers with the contents of the file since we are usingio.Reader
. We need to read the interface once to identify which yarn flavor we have and once more for processing the lock file.io.Reader
cannot be reverted and that is why we store the content in the buffers. -
classic.go
: Most of the code of the old yarn.go is now here. Nothing is changed. Unit tests supports almost 100% coverage. -
berry.go
: For berry flavor we parse the yaml file and then for eachYamlPackage
we make sure that the name is parsed correctly
-
- Added unit tests for yarn v3.
What are the relevant issue numbers?
gitlab-org/gitlab#263358 (closed), gitlab-org/gitlab#351841 (closed)
Does this MR meet the acceptance criteria?
-
Changelog entry added -
Documentation created/updated for GitLab EE, if necessary -
Documentation created/updated for this project, if necessary -
Documentation reviewed by technical writer or follow-up review issue created -
Tests added for this feature/bug -
Job definition updated, if necessary -
Conforms to the code review guidelines -
Conforms to the Go guidelines -
Security reports checked/validated by reviewer
Edited by Nick Ilieskou