Skip to content

Package Registry: Truncate Pypi metadata description field

Radamanthus Batnag requested to merge 431505-twine-push-to-package-registry-fails-with-validation-failed-description-is-too-long-maximum into master

What does this MR do and why?

We recently started uploading more Pypi metadata fields in !131327 (merged).

Unfortunately, because we added a size limit on the description field, packages with a large README file hit this size limit. When a Pypi package with a large README file is uploaded, we return an error response:

Uploading albedo_gscli-0.0.20.dev3+g9502f74-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.6/111.6 kB • 00:00 • 235.3 MB/s
INFO     Response from https://gitlab.com/api/v4/projects/<omitted>/packages/pypi:
         400 Bad Request
INFO     {"message":"Validation failed: Description is too long (maximum is 4000 characters)"}
ERROR    HTTPError: 400 Bad Request from https://gitlab.com/api/v4/projects/<omitted>/packages/pypi
         Bad Request

This MR will truncate the description so that such package can be uploaded, with the truncated description in the metadata.

Screenshots or screen recordings

No UI changes 🌈

How to set up and validate locally

  1. Install the prerequisites
  1. Download this barebones python package
  2. Edit README.md so that it is larger than 4KB. Our size limit is 4000 characters.
  3. Build the package. From the package directory, run python3 -m build
  4. Setup authentication following our guide
  5. Upload the package. From the package directory, run python3 -m twine upload --verbose --repository gitlab dist/*. NOTE: The upload will fail if you try to upload a version that already exists. Delete the existing version in Package Registry, or build and upload a different version.

Expected response:

Uploading ansible-1.0.14.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.4/114.4 kB • 00:00 • 356.0 MB/s
INFO     Response from http://gdk.test:3000/api/v4/projects/34/packages/pypi:                                                                                                                               
         201 Created                                                                                                                                                                                        
INFO     {"message":"201 Created"}
  1. Verify that the package metadata was populated. From a Rails console, run ::Packages::Package.last.pypi_metadatum.description. The metadatum record should have been created, with the description field set to the truncated contents of README.md.

In the master branch, the upload will fail:

Uploading ansible-1.0.15-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.0/114.0 kB • 00:00 • 305.9 MB/s
INFO     Response from http://gdk.test:3000/api/v4/projects/34/packages/pypi:                                                                                                                               
         400 Bad Request                                                                                                                                                                                    
INFO     {"message":"Validation failed: Description is too long (maximum is 4000 characters)"}                                                                                                              
ERROR    HTTPError: 400 Bad Request from http://gdk.test:3000/api/v4/projects/34/packages/pypi                                                                                                              
         Bad Request

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #431505 (closed)

Edited by Radamanthus Batnag

Merge request reports