Enable Pipenv dev dependency scanning

What does this MR do?

• The piplock parser has been registered in the gemnasium-python main package. This parser is capable of scanning and filtering out dev dependencies found in Pipfile.lock files.
• The pipenv builder has been updated so that it no longer returns a pipdeptree graph export. Instead, it returns the path to the Pipfile.lock of the project, so that the project can scan dev dependencies.
  • If a project does contain an existing Pipfile.lock file, the builder will run pipenv sync --dev to ensure that we do not update the lock file when installing the dependencies. See the sync subcommand documentation for details.
  • If a project does not contain an existing Pipfile.lock file, the builder will run pipenv install --dev to ensure that we create a lock file when installing the dependencies. See the install subcommand documentation for details.
• A hack was added to the pipenv builder so that the names used for dependencies does not change when switching from scanning the graph export to scanning the Pipfile.lock.
• Dependencies declared in the [dev-packages] section of a Pipfile.lock are scanned by default. The parser respects the setting declared by DS_INCLUDE_DEV_DEPENDENCIES, so a false value will exclude these dependencies.
• The pipfile-lock test fixture now includes a development dependency to ensure that we skip the direct and indirect dev dependencies when toggling DS_INCLUDE_DEV_DEPENDENCIES to false in image tests.
• The test project has been updated so that dependency scanning on tests/python-pipenv matches the reports in qa/expect/python-pipenv/.
  • Add `pytest` dev-package (gitlab-org/security-products/tests/python-pipenv!91 - merged) - ensures that we respect the DS_INCLUDE_DEV_DEPENDENCIES variable. Without a dev dependency in the Pipfile it's impossible to tell if we have a regression in the exclusion logic.
  • Fix iptables commands, remove Pipfile.lock (gitlab-org/security-products/tests/python-pipenv!92 - merged) - ensures the builder reuses preinstalled dependencies in environments where there is no network connectivity.
  • Add private registry (gitlab-org/security-products/tests/python-pipenv!93 - merged) - ensures the builder can connect to private registries in environments where there is limited network connectivity. Additionally, it also checks that it uses the certificate in the ADDITIONAL_CA_CERTIFICATE_BUNDLE. For evidence see this job.

IMPORTANT (updated)

The move to generating and parsing Pipfile.lock files has impacted the package names in the generated reports i.e. gl-dependency-scanning-report.json and gl-sbom-pypi-pip.cdx.json. This is because Pipenv applies PEP8 standards to the packages fetched (which is also what PyPi does).

The names are currently transformed by the builder to the values previously used. This is a temporary addition until the Rails backend takes care of deduplicating findings generated by either the canonical name or the normalized names.

What are the relevant issue numbers?

• Closes Enable development dependency scanning in pipen... (gitlab-org/gitlab#375505 - closed)
• Closes Dependency Scanning of Pipfile.lock without ins... (gitlab-org/gitlab#299294)
• Relates to Update fingerprinting method to prevent finding... (gitlab-org/gitlab#409495)

Does this MR meet the acceptance criteria?

• Changelog entry added
• Documentation created/updated for GitLab EE, if necessary
• Documentation created/updated for this project, if necessary
• Documentation reviewed by technical writer or follow-up review issue created
• Tests added for this feature/bug
• Job definition updated, if necessary
  • Auto-DevOps template
  • Job definition example
  • CI Templates
• Conforms to the code review guidelines
• Conforms to the Go guidelines
• Security reports checked/validated by reviewer