Discover, share and install PHP dependencies with the GitLab Composer Repository

Problem to solve

Composer, is a dependency manager for PHP that allows you to declare the libraries your project depends on and it will manage (install/update) them for you. PHP developers need a mechanism to share and consume packages and view metadata in one central location.

Intended users

Proposal

Provide support for users coding in PHP by integrating with Composer and allowing developers to share and consume packages alongside their source code and pipelines.

Proposed MVC Scope

Publish dependencies to your GitLab project
Install dependencies using your group-level endpoint.
Authenticate using your GitLab Personal Access Token.
View your dependencies and their respective metadata using the Package Registry UI

Package Naming and Scope

The package name must be unique across the instance to allow easy integration. For the MVP the package name will be claimed by the first use of a given name. For example, when a package named my-package is first pushed to project project-1, then that name is owned by project-1. From then on only users with permission to project P can access the package.

Packages are not uploaded to the Composer Repository, they are linked.

It's important to clarify that Composer doesn't handle uploading packages to the registry as it's only the meta-layer for mapping package names to actual resources (which can be archives or Git references). As Gitlab already hosts ZIP files for branches and tags the only thing that needs to be done here is generating JSON with package information and pointing packages to existing resources.

Further details

Implementation

Composer relies heavily on VCS for versioning by using a naming convention for branches and tags (See composer docs).

A simple way to publish a package would be to simply expose the package archive straight from GitLab by using the Blob Api.

A package could then be published after it's on git by posting to the API:

curl -X POST -F -F 'name=package_name' -F 'branch=my-feature' http://gitlab.com/<project_id>/packages/composer

This would expose the specific branch as a composer package. Or to expose a tag:

curl -X POST -F 'name=package_name' -F 'tag=v1.0' http://gitlab.com/<project_id>/packages/composer

This API call creates the package inside GitLab and then parses the version accordingly based on composer rules.

Composer Package API

Composer package install starts by fetching the Packagist index from https://repo.packagist.org/packages.json. This top-level JSON works as a pagination mechanism by pointing to other indexed JSON files.

Sample index

{
    "metadata-url": "/p2/%package%.json",
    "notify": "https://packagist.org/downloads/%package%",
    "notify-batch": "https://packagist.org/downloads/",
    "packages": [],
    "provider-includes": {
        "p/provider-2013$%hash%.json": {
            "sha256": "c9f9d0aa9b4bc58886635d7768e599bf283c0b6d97ac7e877a9d0858a78903d7"
        },
        ...
        "p/provider-2020-01$%hash%.json": {
            "sha256": "6c3bb1746ab8ddda1503240e416b73d15ee86c50df0d1b39a390b32df78b4c6c"
        },
        "p/provider-archived$%hash%.json": {
            "sha256": "95b7476cfaab2a59b34e1fbc07c1b5eaea0ffab22762f7f3baab102b0cc62822"
        },
        "p/provider-latest$%hash%.json": {
            "sha256": "490b8c4812dd4535ff03c1b378c4f6a5caa855bfa961152cee65104575de60d4"
        }
    },
    "providers-api": "https://packagist.org/providers/%package%.json",
    "providers-url": "/p/%package%$%hash%.json",
    "search": "https://packagist.org/search.json?q=%query%&type=%type%"
}

Index files

The second level index contains a list of URLs for the packages:

GET http://repo.packagist.org/p/provider-2020-01%246c3bb1746ab8ddda1503240e416b73d15ee86c50df0d1b39a390b32df78b4c6c.json

{
  "providers": {
    "0.0.0/laravel-env-shim": {
      "sha256": "3b765076f87ac782e172d802b08d84c6d6c22e7845263c7bbcdce475f3bdf84c"
    },
    ...
    "symphonycms/symphony-2": {
      "sha256": "08bd8ae21aad64fb0f7965a086f0d386523eda3b0a24911b42ee485028f95adf"
    },
  }
}

Package versions file

The 3rd level index file has information about the versions of a specific package:

GET http://repo.packagist.org/p/symphonycms/symphony-2%2408bd8ae21aad64fb0f7965a086f0d386523eda3b0a24911b42ee485028f95adf.json

{
  "packages": {
    "symphonycms/symphony-2": {
      "2.6.0": {
        "name": "symphonycms/symphony-2",
        "description": "Symphony is a PHP & MySQL based CMS that utilises XML and XSLT as its core technologies.",
        "keywords": [],
        "homepage": "http://www.getsymphony.com",
        "version": "2.6.0",
        "version_normalized": "2.6.0.0",
        "license": [
          "MIT"
        ],
        "authors": [
          {
            "name": "Symphony Team",
            "email": "team@getsymphony.com"
          }
        ],
        "source": {
          "type": "git",
          "url": "https://github.com/symphonycms/symphony-2.git",
          "reference": "5a4883563627d6e9f5a941a7e01328d2e84ab00f"
        },
        "dist": {
          "type": "zip",
          "url": "https://api.github.com/repos/symphonycms/symphony-2/zipball/5a4883563627d6e9f5a941a7e01328d2e84ab00f",
          "reference": "5a4883563627d6e9f5a941a7e01328d2e84ab00f",
          "shasum": ""
        },
        "type": "library",
        "time": "2015-03-11T09:23:21+00:00",
        "autoload": {
          "classmap": [
            "symphony/content",
            "symphony/lib",
            "symphony/template",
            "install"
          ],
          "files": [
            "symphony/lib/boot/func.utilities.php",
            "symphony/lib/boot/defines.php",
            "symphony/lib/toolkit/util.validators.php"
          ]
        },
        "uid": 354000
      },
      ...
    }
  }
}

From there the specific package zip file for a given version can be retrieved (in this case straight from github):

GET https://api.github.com/repos/symphonycms/symphony-2/zipball/68f44f0c36ad3345068676bfb8a61c2e6a2e51f4

Package Index Pagination

The packages.json top-level entry point should return a paginated list of packages. For the MVC I believe we could paginate by the package name initial letters like so:

gitlab.com/api/v4/packages/composer/packages/aa.json gitlab.com/api/v4/packages/composer/packages/ab.json ... gitlab.com/api/v4/packages/composer/packages/zz.json

Entry Points

GET gitlab.com/api/v4/packages/composer/packages.json: Instance level, paginated top level index for the list of available packages.

GET gitlab.com/api/v4/packages/composer/packages/:page.json: The paginated list of package names. Returns the list of packages contained in a specific :page.

GET gitlab.com/api/v4/projects/:id/packages/composer/*package_name.json: Provides the package metadata json with the data about the package versions.

GET gitlab.com/api/v4/projects/:id/packages/composer/*package_name/-/*file_name: Endpoint to download a specific package file.

POST gitlab.com/api/v4/projects/:id/packages/composer: Endpoint to upload package files.

Permissions and Security

Access to packages should follow standard project read/write permissions.

Documentation

Add the Composer Repository to https://docs.gitlab.com/ee/user/packages/
Screenshots should also be updated to reflect that Composer is now an option

Availability & Testing

What does success look like, and how can we measure that?

Success looks like we enable PHP developers to use their GitLab project as a Composer Repository and drive usage of the Package Registry and CI/CD by giving them a new GitLab workflow they can use.

Metrics

Our North Star Metric of number of packages published/installed will suffice to measure adoption.
We can also monitor page views and the number of times someone uses the quick setup/install commands in the UI.

Edited Jul 09, 2020 by Tim Rizzi