Document requirement for ZStandard compression

What does this MR do and why?

Document requirement to support ZStandard compression for Debian binary packages.

Since dpkg 1.21.18 compression zstd is supported. It is used by default since Ubuntu 21.10. The Gitlab Debian package registry will not recognize such binary packages as valid and fails with

Invalid Package: failed metadata extraction

Document requirement to support zstd.

#422584

Screenshots or screen recordings

How to set up and validate locally

• Enable Debian API for this project by an Administrator until #337288 is resolved.
• See sample project https://gitlab.com/pmhahn/debian-package-registry

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

mentioned in issue #422584

Hey @pmhahn!

Welcome to our community! We're excited to have you here, and can't wait to review this first MR with you!

Thank you for your contribution to GitLab. Please refer to the contribution documentation for an overview of the process.

Did you know about our community forks? Working from there will make your contribution process easier. Please check it out!

When you're ready for a first review, post @gitlab-bot ready. If you know a relevant reviewer(s) (for example, someone that was involved in a related issue), you can also assign them directly with @gitlab-bot ready @user1 @user2.

At any time, if you need help, feel free to post @gitlab-bot help or initiate a mentor session on Discord. Read more on how to get help.

This message was generated automatically. You're welcome to improve it.

added 1st contribution Community contribution workflowin dev labels

assigned to @pmhahn

@gitlab-bot ready @sathieu

added workflowready for review label and removed workflowin dev label

requested review from @sathieu

@sathieu, this Community contribution is ready for review.

• Do you have capacity and domain expertise to review this? We are mindful of your time, so if you are not able to take this on, please re-assign to one or more other reviewers.
• Add the workflowin dev label if the merge request needs action from the author.

This message was generated automatically. You're welcome to improve it.

added linked-issue label

added 1 commit

• c63ccf78 - Support Debian package registry zstd compression

Compare with previous version

added workflowin dev label and removed workflowready for review label

added devopspackage grouppackage registry labels

added sectionci label

added typemaintenance label

added backend label

added maintenancerefactor label

changed milestone to %16.4

Actually my patch is wrong (or incomplete):

• I changed app/services/packages/debian/extract_metadata_service.rb # file_type_source, which allows ZStandard for source packages. The compression algorithm for source packages is controlled by dpkg-source -Z….
• but app/services/packages/debian/extract_deb_metadata_service.rb # cmd uses Popen(…) to invoke the external dpkg --field binary to extract the fields from any binary package. Gitlab.config.packages.dpkg_deb_path defaults to /usr/bin/dpkg-deb, so it is the version from our own on-premise installation (currently Debian 10.13 Buster). Even the version from Debian 11.7 Bullseye is too old to support ZStandard. Only the version from Debian 12 Bookworm supports it (or the equivalent from Ubuntu). The compression algorithm for binary packages is controlled by dpkg-deb -Z….

As I'm a Debian Developer myself I also did some more research on dpkg and the supported compression algorithms:

compress	since	deprecation	Year	Debian	Ubuntu
none	1.10.24		2004	3.1 Sarge
gzip
bzip2	1.10.24	1.18.11	2004	3.1 Sarge
lzma	1.13.25	1.18.11	2007	4 Etch
xz	1.15.6		2010	6 Squeeze
zstd	1.21.18		2023	12 Bookworm	18.04 Bionic

bzip2 and lzma have been deprecated by Debian for building binary packages, but extracting existing packages and source files is still supported. But as you can still use older versions of dpkg itself to build Debian packages, Gitlab should still support these deprecated algorithms as Debian does for extraction.

So support for ZStandard depends on the version of Debian package dpkg installed on the Gitlab host/cluster/environment: Ubuntu supports it since 2018, Debian only since 2023. I see multiple options:

1. Document that requirement: Support for "ZStandard" compressed Debian binary packages depends on the version of dpkg-deb available to Gitlab; the path to the binary can be configured in gitlab.yml via packages.dpkg_deb_path.
2. Ship a new enough version of dpkg with Gitlab, but that will require shipping several shared libraries like libz, liblzma, libbz2, libzstd, …
3. Find a replacement for calling the external dpkg-deb by Popen(): I'm no ruby developer myself but maybe there is one ruby gem to work with Debian binary packages natively; for Python there is PyPI:python-debian. The Debian binary package format is quiet simple, but handling the different compression formats may be the tricky part.

added 5 commits

• 06400ff5 - Document requirement for Debian compression zstd
• 3ee41ddc - Document Debian codename is string
• c866c222 - Document Debian architectures/components are string arrays
• a513711e - Document Debian codename vs. suite
• 99d8b211 - Document Debian apt usage

Compare with previous version

added 5 commits

• 7f7f889a - Document requirement for Debian compression zstd
• e79251ab - Document Debian codename is string
• a1d2dec4 - Document Debian architectures/components are string arrays
• b88f8ec7 - Document Debian codename vs. suite
• 809b03c9 - Document Debian apt usage

Compare with previous version

resolved all threads

• drop initial patch because it was wrong
• document requirement for supporting ZStandard
• update documentation related to Debian package registry in Gitlab

added workflowready for review label and removed workflowin dev label

requested review from @dmeshcharakou

Hi @phillipwells! Please review this documentation merge request. This message was generated automatically. You're welcome to improve it.

added documentation twtriaged labels

requested review from @phillipwells

@sathieu @phillipwells @dmeshcharakou, this Community contribution is ready for review.

• Do you have capacity and domain expertise to review this? We are mindful of your time, so if you are not able to take this on, please re-assign to one or more other reviewers.
• Add the workflowin dev label if the merge request needs action from the author.

This message was generated automatically. You're welcome to improve it.

marked this merge request as draft

marked this merge request as ready

changed the description

I just verified it again that zstd it not yet supported for source packages:

docker run --rm -ti -v $PWD:/work -w /work ubuntu:22.04
apt update
apt install --no-install-recommends dpkg-dev
dpkg --version
# Debian 'dpkg' package management program version 1.21.1 (amd64).
dpkg-source -b hello-1.1
# dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../hello_1.1.orig.tar.{bz2,gz,lzma,xz}
cat hello-1.1/debian/source/format 
# 3.0 (quilt)
ls -1d hello*
# hello-1.1
# hello_1.1.orig.tar.zst

• app/services/packages/debian/extract_metadata_service.rb must not be updated and as such spec/services/packages/debian/extract_metadata_service_spec.rb needs not to be updated.
• only doc/user/packages/debian_repository/index.md needs to be updated to state the requirement for a new dpkg-deb binary.

@pmhahn Thanks for the contribution! I've left a handful of suggestions from a Technical Writing perspective. Once you've given them a review, feel free to @mention me so I can take a final look

added Technical Writing docsimprovement labels

removed review request for @phillipwells

added 5 commits

• 464fbb5a - Document requirement for Debian compression zstd
• 7fc1d601 - Document Debian codename is string
• fa9cc814 - Document Debian architectures/components are string arrays
• 4e3e5f28 - Document Debian codename vs. suite
• abe3b6f5 - Document Debian apt usage

Compare with previous version

resolved all threads

Applied all suggestions from @phillipwells , squashed and --force pushed.

changed title from feat(Debian): support zstd compression to Document requirement to support ZStandard compression for Debian binary packages

changed the description

requested review from @phillipwells

added 1 commit

• ea4e4787 - fixup! Document requirement for Debian compression zstd

Compare with previous version

marked this merge request as draft from univention/gitlab@ea4e4787

Applied all suggestions from @phillipwells , squashed and --force pushed.

marked this merge request as ready

added 5 commits

• 1a70999c - Document requirement for Debian compression zstd
• 3edfd4b1 - Document Debian codename is string
• dd4201f7 - Document Debian architectures/components are string arrays
• 41bc9703 - Document Debian codename vs. suite
• 0cbc3005 - Document Debian apt usage

Compare with previous version

approved this merge request

removed review request for @dmeshcharakou

resolved all threads

Everything looks good from my perspective, so I'll approve and set to merge! Thanks again, @pmhahn

approved this merge request

enabled an automatic merge when the pipeline for 7062249c succeeds

	1 Message
	This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

Documentation review

The following files require a review from a technical writer:

• doc/api/packages/debian_group_distributions.md (Link to current live version)
• doc/api/packages/debian_project_distributions.md (Link to current live version)
• doc/development/packages/debian_repository.md (Link to current live version)
• doc/user/packages/debian_repository/index.md (Link to current live version)

The review does not need to block merging this merge request. See the:

• Metadata for the *.md files that you've changed. The first few lines of each *.md file identify the stage and group most closely associated with your docs change.
• The Technical Writer assigned for that stage and group.
• Documentation workflows for information on when to assign a merge request for review.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

changed title from Document requirement to support ZStandard compression for Debian binary packages to Document requirement for ZStandard compression

merged

@pmhahn, how was your code review experience with this merge request? Please tell us how we can continue to iterate and improve:

1. React with a or a on this comment to describe your experience.
2. Create a new comment starting with @gitlab-bot feedback below, and leave any additional feedback you have for us in the comment.

Interested in learning more tips and tricks to solve your next challenge faster? Subscribe to the GitLab Community Newsletter for contributor-focused content and opportunities to level up.

Thanks for your help!

This message was generated automatically. You're welcome to improve it.

@pmhahn, congratulations for getting your first MR merged

If this is your first MR against a GitLab project, we'd like to invite and encourage you to self-nominate yourself for First MR Merged swag prize here.

Thank you again for contributing, what's your next contribution going to be?

This message was generated automatically. You're welcome to improve it.

mentioned in commit 555d7429

added workflowstaging-canary label and removed workflowready for review label

added workflowcanary label and removed workflowstaging-canary label

added workflowstaging label and removed workflowcanary label

added workflowproduction label and removed workflowstaging label

added workflowpost-deploy-db-staging label and removed workflowproduction label

added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label

added releasedcandidate label

added releasedpublished label and removed releasedcandidate label