Add Python 3 to the retire.js analyzer, make Python 2 optional

Problem to solve

Follow-up to https://gitlab.com/gitlab-org/gitlab-ee/issues/11413

When running npm install, the retire.js analyzer plugin is unable to install native modules/addons(packages that build with node-gyp).

We should also ensure that the other necessary prerequisites are installed for node-gyp: https://github.com/nodejs/node-gyp#on-unix

Intended users

Persona: Software developer

Further details

This happens because python is not the part of node:11-alpine base image upon which the retire.js analyzer plugin image is built.

Because of this, we're unable to run npm install for projects with native modules/addons dependencies like Gitter.

https://gitlab.com/gitlab-org/gitter/webapp/-/jobs/235354697

Installing dependencies...

> heapdump@0.3.12 install /tmp/app/node_modules/heapdump
> node-gyp rebuild

gyp ERR! configure error 
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at PythonFinder.failNoPython (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:484:19)
gyp ERR! stack     at PythonFinder.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:406:16)
gyp ERR! stack     at F (/usr/local/lib/node_modules/npm/node_modules/which/which.js:68:16)
gyp ERR! stack     at E (/usr/local/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqCallback.oncomplete (fs.js:158:21)

Proposal

  • leverage these variables in the gemnasium-python analyzer
  • Add python to the Dockerfile of the retire.js analyzer plugin image.
  • Allow users to set Python 2 or 3 as part of this solution. We can implement a DS_PYTHON_VERSION variable similar to LM_PYTHON_VERSION done in (https://gitlab.com/gitlab-org/security-products/license-management/merge_requests/36). Default version should be 3.
  • Add DS_PYTHON_VERSION to our vendored template to pass them down to the analyzers.

Permissions and Security

No special permissions

Documentation

  • Add the new variable in Dependency Scanning documentation: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html#available-variables. We probably need to specify that only the retire.js analyzer is supporting this option (and not the gemnasium-python one).

Testing

  • Native modules/addons dependency should be added to js-npm test project
  • Native modules/addons dependency should be added to js-yarn test project

What does success look like, and how can we measure that?

The number of customers that have problems with npm-based projects with native modules/addons dependencies.

What is the type of buyer?

  • Internal customers
  • GitLab Ultimate users

Links / references

cc @vzagorodny

Edited Jul 22, 2019 by Can Eldem
Assignee Loading
Time tracking Loading