Improve performance for composer v2 clients (!55169) · Merge requests · GitLab.org / GitLab

Giorgenes Gelatti requested to merge 290288-disable-composer-shas-for-v2 into master Feb 25, 2021

🏘 Context

The GitLab package registry supports the Composer package manager for php packages. Recently, Composer released V2. The big change for registry providers is the new metadata-url endpoint. There are two parts, first, when a user installs a composer package, an "index" request is made. It includes some metadata about all packages in the registry, including a providers-url that is used by the Composer client to fetch packages. V2 adds a metadata-url to the payload. It is similar in that it is a URL template with a placeholder for the package name that the Composer client will then use to make requests for the package.

When the package is requested via the metadata-url, it makes a request to /api/v4/groups/:id/-/packages/composer/p2/*package_name.

All of the changes described thus far were added in !54906 (merged).

Now, we see that we return both the metadata-url and providers information for any incoming request. However, there is a problem with the V1 providers-url version in that it also needs to include a digest (sha256) of all packages within the index. The digest is calculated on the fly, so if we have a group that has thousands and thousands of packages distributed throughout hundreds of subgroups, collecting them and calculating a single digest from them all is slow and not a great UX. We have been addressing that problem in #290288 (closed).

The good news is, Composer V2 doesn't need that digest anymore, so we can leave it out of the response as long as the user is using Composer V2!

🔎 What does this MR do?

Conditionally removes provider SHA's from the composer index if the composer client is version. We do this by checking the user-agent header. This allows Composer v2 clients to experience a much faster and significantly improved response time since computing the SHA's can be very slow.

📸 Screenshots (strongly suggested)

This compares the current behavior vs the new behavior when using composer V2. From the user perspective, there is no change. The installation is successful regardless of if the providers-url and provider-includes is included in the payload.

Current behavior on master

→ composer update -vvv Reading ./composer.json (/Users/steveabrams/workspace/playground/composer/composer-local-install/composer.json) Loading config file /Users/steveabrams/.composer/config.json Loading config file /Users/steveabrams/.composer/auth.json Loading config file ./composer.json (/Users/steveabrams/workspace/playground/composer/composer-local-install/composer.json) Checked CA file /private/etc/ssl/cert.pem: valid

Running 2.0.11 (2021-02-24 14:57:23) with PHP 7.3.11 on Darwin / 19.6.0 Loading composer repositories with package information Warning: Accessing gdk.test over http which is an insecure protocol. Using HTTP basic authentication with username "token" Downloading http://gdk.test:3001/api/v4/group/153/-/packages/composer/packages.json [200] http://gdk.test:3001/api/v4/group/153/-/packages/composer/packages.json Writing /Users/steveabrams/.composer/cache/repo/http---gdk.test-3001-api-v4-group-153---packages-composer-packages.json/packages.json into cache Downloading http://gdk.test:3001/api/v4/group/153/-/packages/composer/p2/foo/composer-test.json [200] http://gdk.test:3001/api/v4/group/153/-/packages/composer/p2/foo/composer-test.json Writing /Users/steveabrams/.composer/cache/repo/http---gdk.test-3001-api-v4-group-153---packages-composer-packages.json/provider-foo~composer-test.json into cache Updating dependencies Generating rules Resolving dependencies through SAT Looking at all rules.

Dependency resolution completed in 0.000 seconds Analyzed 86 packages to resolve dependencies Analyzed 86 rules to resolve dependencies Lock file operations: 1 install, 0 updates, 0 removals Installs: foo/composer-test:1.0.0 - Locking foo/composer-test (1.0.0) Writing lock file Installing dependencies from lock file (including require-dev) Reading ./composer.lock (/Users/steveabrams/workspace/playground/composer/composer-local-install/composer.lock) Package operations: 1 install, 0 updates, 0 removals Installs: foo/composer-test:1.0.0 - Downloading foo/composer-test (1.0.0) Downloading http://gdk.test:3001/api/v4/projects/61/packages/composer/archives/foo/composer-test.zip?sha=98298a129ca79d3c1c55a6651993ac01109e34ae [200] http://gdk.test:3001/api/v4/projects/61/packages/composer/archives/foo/composer-test.zip?sha=98298a129ca79d3c1c55a6651993ac01109e34ae Writing /Users/steveabrams/.composer/cache/files/foo/composer-test/587bfaa79eec55a422b392a5529929f8e5b816f3.zip into cache from /Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/tmp-9422a905db2c04fd30ecf63fed33a0e7.zip - Installing foo/composer-test (1.0.0): Extracting archive Executing async command (CWD): unzip -qq -o '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/tmp-9422a905db2c04fd30ecf63fed33a0e7.zip' -d '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/54c7863d' Executing command (CWD): rm -rf '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/foo/composer-test' Executing command (CWD): rm -rf '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/54c7863d' Executing command (CWD): rm -rf '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/' Generating autoload files

With the V2 Conditional (this MR)

Dependency resolution completed in 0.000 seconds Analyzed 86 packages to resolve dependencies Analyzed 86 rules to resolve dependencies Lock file operations: 1 install, 0 updates, 0 removals Installs: foo/composer-test:1.0.0 - Locking foo/composer-test (1.0.0) Writing lock file Installing dependencies from lock file (including require-dev) Reading ./composer.lock (/Users/steveabrams/workspace/playground/composer/composer-local-install/composer.lock) Package operations: 1 install, 0 updates, 0 removals Installs: foo/composer-test:1.0.0 - Downloading foo/composer-test (1.0.0) Downloading http://gdk.test:3001/api/v4/projects/61/packages/composer/archives/foo/composer-test.zip?sha=98298a129ca79d3c1c55a6651993ac01109e34ae [200] http://gdk.test:3001/api/v4/projects/61/packages/composer/archives/foo/composer-test.zip?sha=98298a129ca79d3c1c55a6651993ac01109e34ae Writing /Users/steveabrams/.composer/cache/files/foo/composer-test/587bfaa79eec55a422b392a5529929f8e5b816f3.zip into cache from /Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/tmp-05a43eee8212fe18fb0aa3e1f70bae6c.zip - Installing foo/composer-test (1.0.0): Extracting archive Executing async command (CWD): unzip -qq -o '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/tmp-05a43eee8212fe18fb0aa3e1f70bae6c.zip' -d '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/0ceca33d' Executing command (CWD): rm -rf '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/foo/composer-test' Executing command (CWD): rm -rf '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/0ceca33d' Executing command (CWD): rm -rf '/Users/steveabrams/workspace/playground/composer/composer-local-install/vendor/composer/' Generating autoload files

☑ Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
[-] Tested in all supported browsers
[-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

[-] Label as security and @ mention @gitlab-com/gl-security/appsec
[-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
[-] Security reports checked/validated by a reviewer from the AppSec team

Edited Mar 16, 2021 by Steve Abrams

Improve performance for composer v2 clients