Add support for PyPI packages

📡 Context

We want to be able to import PyPI packages. See Use pipelines to import PyPI packages (gitlab-org/gitlab#389339 - closed) for more details.

Add all the necessary pieces to support importing PyPI packages.

🤔 What does this MR do?

• add the pypi type registry.
  • add the required scripts (steps) to pull a package from registry A and publish it to B.
• add the related tests.
• add the related documentation.

📺 Screens

⚙ How to validate this locally?

The suggested way to test this is to run the tool locally. That will generate a pipeline config, then put that config in a .gitlab-ci.yml file and start a pipeline.

Have a config file with (adjust as necessary):

config file

from_artifactory:
  type: pypi
  source:
    url: https://tenandio.jfrog.io/artifactory/api/pypi/pypi_test/simple
    credentials:
      username: $ARTIFACTORY_USERNAME
      token: $ARTIFACTORY_TOKEN
  destination: &destination
    url: https://gitlab.com/api/v4/projects/45821687/packages/pypi
    credentials:
      token: $GITLAB_TOKEN
      username: $GITLAB_USERNAME
  packages:
    "mypkg": 0.1

from_gitlab:
  type: pypi
  source:
    url: https://gitlab.com/api/v4/projects/31037479/packages/pypi/simple
    credentials:
      token: $GITLAB_TOKEN
      username: $GITLAB_USERNAME
  destination: *destination
  packages:
    "Transcof": 8.7.5
    "Redhold": 5.1.5
    "Fixflex": 2.6.7
    "Vagram": 3.0.8

from_pypi_org:
  type: pypi
  source:
    url: https://pypi.python.org/simple
  destination: *destination
  packages:
    "graphistry": 0.29.2
    "twine": 4.0.2

You will need a few CI variables. GitLab tokens can be PATs.

Run the tool:

$ go run main.go generate

Now, in a GitLab project, copy the contents of child_pipeline.yml in a .gitlab-ci.yml file.

Start a pipeline and watch the PyPI packages being imported. 🍿