Add support for Debian packages using compression zstd

Summary

Debian packages support different compression algorithms: none, gzip, xz, zstd. They have been added over the years and the default is also changed from time to time.

Since Ubuntu 21.10 now defaults to zstd, which Gitlab Debian Package registry does not seem to handle yet.

  • Uploading .deb files using zstd does not produce any error
  • but the package will be named debian-temporary-package
  • it does show "⚠️ Error publishing"
  • the tool-tip indicates "Invalid Package: failed metadata extraction".

Steps to reproduce

  1. Build a Debian package with dpkg-deb -Zzstd.
  2. Upload it to the Debian package registry (DPR) using dput or similar
  3. Navigate to Package Registry

Example Project

https://gitlab.com/pmhahn/debian-package-registry

The Debian API must be enabled for this project by an Administrator until #337288 is resolved.

What is the current bug behavior?

Debian packages compressed with zstd will show up as debian-temporary-package

What is the expected correct behavior?

Debian packages compressed with zstd

Relevant logs and/or screenshots

image

$ LC_ALL=C LANG=C LANGUAGE=C dpkg -I deb-zstd_1.0_all.deb
dpkg-deb: error: archive 'deb-zstd_1.0_all.deb' uses unknown compression for member 'control.tar.zst', giving up
$ ar t deb-zstd_1.0_all.deb
debian-binary
control.tar.zst
            ^^^
data.tar.zst
         ^^^

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Debian 10
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.0.6p216
Gem Version:    3.4.14
Bundler Version:2.4.16
Rake Version:   13.0.6
Redis Version:  7.0.12
Sidekiq Version:6.5.7
Go Version:     unknown

GitLab information
Version:        16.2.4-ee
Revision:       9544e5451d7
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.11
URL:            https://git.knut.univention.de
HTTP Clone URL: https://git.knut.univention.de/some-group/some-project.git
SSH Clone URL:  git@git.knut.univention.de:some-group/some-project.git
Elasticsearch:  yes
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:


GitLab Shell
Version:        14.23.0
Repository storages:


    

  • default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
    •

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/7


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... Checking gitlabprojects@univention.de
Checking gitlabprojects@univention.de
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
3/2 ... yes
3/3 ... yes
3/4 ... yes
3/5 ... yes
9/10 ... yes
4/16 ... yes
3/17 ... yes
139/18 ... yes
139/20 ... yes
3/21 ... yes
3/22 ... yes
139/23 ... yes
3/25 ... yes
139/31 ... yes
3/33 ... yes
3/34 ... yes
3/35 ... yes
10/38 ... yes
34/39 ... yes
10/40 ... yes
11/42 ... yes
16/44 ... yes
39/46 ... yes
139/47 ... yes
39/48 ... yes
139/49 ... yes
3/50 ... yes
34/51 ... yes
118/53 ... yes
41/54 ... yes
41/55 ... yes
41/56 ... yes
39/57 ... yes
4/59 ... yes
4/60 ... yes
4/61 ... yes
11/62 ... yes
11/63 ... yes
16/64 ... yes
3/65 ... yes
46/66 ... yes
46/67 ... yes
46/68 ... yes
46/69 ... yes
39/70 ... yes
46/71 ... yes
11/73 ... yes
39/74 ... yes
19/76 ... yes
107/77 ... yes
39/78 ... yes
139/79 ... yes
118/80 ... yes
139/81 ... yes
51/82 ... yes
11/83 ... yes
46/84 ... yes
107/85 ... yes
46/86 ... yes
46/87 ... yes
158/89 ... yes
51/90 ... yes
158/91 ... yes
17/93 ... yes
39/94 ... yes
158/96 ... yes
39/98 ... yes
46/99 ... yes
107/100 ... yes
46/101 ... yes
139/102 ... yes
118/104 ... yes
11/105 ... yes
9/107 ... yes
9/109 ... yes
118/110 ... yes
17/112 ... yes
46/113 ... yes
39/115 ... yes
11/116 ... yes
11/117 ... yes
118/119 ... yes
118/120 ... yes
39/122 ... yes
17/123 ... yes
118/136 ... yes
17/137 ... yes
11/139 ... yes
46/140 ... yes
11/141 ... yes
46/142 ... yes
118/144 ... yes
107/145 ... yes
118/146 ... yes
68/148 ... yes
11/149 ... yes
274/150 ... yes
7/151 ... yes
39/152 ... yes
39/153 ... yes
107/155 ... yes
39/156 ... yes
46/158 ... yes
68/159 ... yes
39/160 ... yes
4/161 ... yes
31/162 ... yes
39/163 ... yes
31/164 ... yes
139/165 ... yes
68/167 ... yes
39/171 ... yes
11/172 ... yes
51/173 ... yes
68/174 ... yes
51/175 ... yes
68/176 ... yes
4/177 ... yes
46/178 ... yes
11/179 ... yes
68/180 ... yes
51/181 ... yes
7/182 ... yes
139/183 ... yes
72/184 ... yes
51/185 ... yes
20/186 ... yes
4/187 ... yes
8/188 ... yes
14/189 ... yes
46/191 ... yes
7/192 ... yes
118/193 ... yes
51/194 ... yes
46/195 ... yes
118/196 ... yes
19/197 ... yes
72/199 ... yes
46/200 ... yes
11/202 ... yes
16/204 ... yes
68/205 ... yes
20/206 ... yes
46/207 ... yes
16/209 ... yes
151/210 ... yes
41/211 ... yes
68/212 ... yes
68/213 ... yes
9/214 ... yes
71/215 ... yes
139/216 ... yes
71/217 ... yes
46/218 ... yes
88/219 ... yes
7/220 ... yes
19/221 ... yes
86/224 ... yes
7/225 ... yes
34/226 ... yes
19/228 ... yes
151/229 ... yes
4/230 ... yes
39/231 ... yes
107/232 ... yes
39/234 ... yes
86/238 ... yes
139/239 ... yes
46/240 ... yes
71/241 ... yes
19/242 ... yes
118/244 ... yes
14/246 ... yes
107/247 ... yes
107/248 ... yes
51/250 ... yes
54/251 ... yes
10/254 ... yes
39/255 ... yes
20/257 ... yes
17/258 ... yes
39/259 ... yes
19/261 ... yes
107/262 ... yes
129/263 ... yes
129/264 ... yes
232/265 ... yes
129/266 ... yes
129/267 ... yes
1076/269 ... yes
129/270 ... yes
14/271 ... yes
72/272 ... yes
71/273 ... yes
118/275 ... yes
129/276 ... yes
68/277 ... yes
68/278 ... yes
17/279 ... yes
71/285 ... yes
68/286 ... yes
17/289 ... yes
54/290 ... yes
71/291 ... yes
128/292 ... yes
125/294 ... yes
71/296 ... yes
158/298 ... yes
7/300 ... yes
125/302 ... yes
105/303 ... yes
129/308 ... yes
217/309 ... yes
131/310 ... yes
54/311 ... yes
71/314 ... yes
134/315 ... yes
54/317 ... yes
86/318 ... yes
46/322 ... yes
138/324 ... yes
125/325 ... yes
138/326 ... yes
138/327 ... yes
3/328 ... yes
128/329 ... yes
125/330 ... yes
125/331 ... yes
54/332 ... yes
7/334 ... yes
148/336 ... yes
147/337 ... yes
3/338 ... yes
125/341 ... yes
138/342 ... yes
139/343 ... yes
150/346 ... yes
134/350 ... yes
107/351 ... yes
39/354 ... yes
9/355 ... yes
125/357 ... yes
134/358 ... yes
158/359 ... yes
158/360 ... yes
7/362 ... yes
151/365 ... yes
9/366 ... yes
125/367 ... yes
125/368 ... yes
139/369 ... yes
71/370 ... yes
129/371 ... yes
158/372 ... yes
125/373 ... yes
125/374 ... yes
125/375 ... yes
206/376 ... yes
86/379 ... yes
131/380 ... yes
219/381 ... yes
217/387 ... yes
139/388 ... yes
71/389 ... yes
221/391 ... yes
167/392 ... yes
167/395 ... yes
222/396 ... yes
219/397 ... yes
180/398 ... yes
170/399 ... yes
170/402 ... yes
46/403 ... yes
171/404 ... yes
174/405 ... yes
125/406 ... yes
222/408 ... yes
120/409 ... yes
180/410 ... yes
221/411 ... yes
47/412 ... yes
183/413 ... yes
171/414 ... yes
171/415 ... yes
10/417 ... yes
151/418 ... yes
125/419 ... yes
171/421 ... yes
222/423 ... yes
86/425 ... yes
151/427 ... yes
220/430 ... yes
86/431 ... yes
125/436 ... yes
171/437 ... yes
86/438 ... yes
199/439 ... yes
203/440 ... yes
203/441 ... yes
47/442 ... yes
208/444 ... yes
208/445 ... yes
134/449 ... yes
125/450 ... yes
3/451 ... yes
71/452 ... yes
139/453 ... yes
222/454 ... yes
222/455 ... yes
125/456 ... yes
219/457 ... yes
217/458 ... yes
212/459 ... yes
212/460 ... yes
222/463 ... yes
214/464 ... yes
86/465 ... yes
137/466 ... yes
219/467 ... yes
125/468 ... yes
127/471 ... yes
206/472 ... yes
214/473 ... yes
226/475 ... yes
128/476 ... yes
151/477 ... yes
51/478 ... yes
125/479 ... yes
151/480 ... yes
230/481 ... yes
127/482 ... yes
230/483 ... yes
125/484 ... yes
230/485 ... yes
217/486 ... yes
237/488 ... yes
7/489 ... yes
241/490 ... yes
241/491 ... yes
241/492 ... yes
230/493 ... yes
86/494 ... yes
242/495 ... yes
242/496 ... yes
245/497 ... yes
246/498 ... yes
247/499 ... yes
247/500 ... yes
257/501 ... yes
249/502 ... yes
86/503 ... yes
257/504 ... yes
249/505 ... yes
39/507 ... yes
180/509 ... yes
261/510 ... yes
261/511 ... yes
261/512 ... yes
261/513 ... yes
261/514 ... yes
261/515 ... yes
240/516 ... yes
199/518 ... yes
223/519 ... yes
266/521 ... yes
266/522 ... yes
249/523 ... yes
268/524 ... yes
271/525 ... yes
273/526 ... yes
39/527 ... yes
274/528 ... yes
274/529 ... yes
240/530 ... yes
289/534 ... yes
17/535 ... yes
107/536 ... yes
274/537 ... yes
51/538 ... yes
125/539 ... yes
274/540 ... yes
153/541 ... yes
214/542 ... yes
266/544 ... yes
249/545 ... yes
274/546 ... yes
298/547 ... yes
300/549 ... yes
300/550 ... yes
217/551 ... yes
308/553 ... yes
308/554 ... yes
308/555 ... yes
214/556 ... yes
266/557 ... yes
19/558 ... yes
86/559 ... yes
249/560 ... yes
212/561 ... yes
273/562 ... yes
125/563 ... yes
298/564 ... yes
298/565 ... yes
249/567 ... yes
133/568 ... yes
86/569 ... yes
139/570 ... yes
138/571 ... yes
125/573 ... yes
72/574 ... yes
274/575 ... yes
325/577 ... yes
151/578 ... yes
39/580 ... yes
332/581 ... yes
332/582 ... yes
171/583 ... yes
54/584 ... yes
308/585 ... yes
125/586 ... yes
139/587 ... yes
325/588 ... yes
332/589 ... yes
7/590 ... yes
240/591 ... yes
300/593 ... yes
300/594 ... yes
300/595 ... yes
300/596 ... yes
98/597 ... yes
31/598 ... yes
199/599 ... yes
46/600 ... yes
138/601 ... yes
300/602 ... yes
223/603 ... yes
9/604 ... yes
226/605 ... yes
4/606 ... yes
353/607 ... yes
240/608 ... yes
151/611 ... yes
125/612 ... yes
104/613 ... yes
46/615 ... yes
51/616 ... yes
240/617 ... yes
104/618 ... yes
51/619 ... yes
17/620 ... yes
68/621 ... yes
15/622 ... yes
300/624 ... yes
127/625 ... yes
3/626 ... yes
240/630 ... yes
72/631 ... yes
128/632 ... yes
219/633 ... yes
31/634 ... yes
128/635 ... yes
128/636 ... yes
158/637 ... yes
887/639 ... yes
128/640 ... yes
151/641 ... yes
31/642 ... yes
72/643 ... yes
353/644 ... yes
46/645 ... yes
217/646 ... yes
919/648 ... yes
107/649 ... yes
46/650 ... yes
921/653 ... yes
209/654 ... yes
209/656 ... yes
921/657 ... yes
20/659 ... yes
912/660 ... yes
921/662 ... yes
921/663 ... yes
921/664 ... yes
209/665 ... yes
3/666 ... yes
912/667 ... yes
332/668 ... yes
46/669 ... yes
18/670 ... yes
11/671 ... yes
139/672 ... yes
86/673 ... yes
921/674 ... yes
125/675 ... yes
46/676 ... yes
51/678 ... yes
11/679 ... yes
226/680 ... yes
130/681 ... yes
39/683 ... yes
921/684 ... yes
912/685 ... yes
7/686 ... yes
86/687 ... yes
921/688 ... yes
921/689 ... yes
353/690 ... yes
68/694 ... yes
995/695 ... yes
274/696 ... yes
237/697 ... yes
332/698 ... yes
332/699 ... yes
1007/700 ... yes
46/701 ... yes
128/702 ... yes
274/703 ... yes
11/704 ... yes
11/705 ... yes
1023/706 ... yes
46/707 ... yes
46/708 ... yes
4/709 ... yes
274/710 ... yes
39/711 ... yes
139/712 ... yes
139/713 ... yes
1014/714 ... yes
139/715 ... yes
146/716 ... yes
146/717 ... yes
887/718 ... yes
240/719 ... yes
1083/720 ... yes
214/721 ... yes
7/722 ... yes
237/723 ... yes
39/724 ... yes
353/725 ... yes
105/726 ... yes
1056/727 ... yes
151/728 ... yes
359/729 ... yes
102/730 ... yes
134/731 ... yes
134/732 ... yes
134/733 ... yes
274/734 ... yes
1056/735 ... yes
217/736 ... yes
1051/737 ... yes
353/739 ... yes
1056/740 ... yes
31/741 ... yes
1072/742 ... yes
139/743 ... yes
240/744 ... yes
1220/745 ... yes
237/746 ... yes
352/747 ... yes
11/748 ... yes
1056/749 ... yes
274/750 ... yes
128/751 ... yes
39/752 ... yes
11/753 ... yes
11/754 ... yes
3/755 ... yes
1220/756 ... yes
3/757 ... yes
11/762 ... yes
11/763 ... yes
11/764 ... yes
11/765 ... yes
11/766 ... yes
11/767 ... yes
11/768 ... yes
1130/772 ... yes
11/773 ... yes
11/774 ... yes
11/775 ... yes
11/776 ... yes
11/777 ... yes
240/778 ... yes
274/779 ... yes
1220/780 ... yes
1130/781 ... yes
51/782 ... yes
11/783 ... yes
47/784 ... yes
226/785 ... yes
31/786 ... yes
139/787 ... yes
1056/788 ... yes
139/789 ... yes
1130/790 ... yes
1161/793 ... yes
1220/794 ... yes
17/796 ... yes
1056/797 ... yes
105/798 ... yes
1220/799 ... yes
199/801 ... yes
1056/802 ... yes
1173/803 ... yes
1220/804 ... yes
921/807 ... yes
138/808 ... yes
1056/809 ... yes
1056/810 ... yes
1189/811 ... yes
1130/812 ... yes
17/813 ... yes
51/814 ... yes
995/815 ... yes
289/816 ... yes
1198/817 ... yes
289/818 ... yes
289/819 ... yes
1130/820 ... yes
125/821 ... yes
1014/822 ... yes
1130/823 ... yes
912/825 ... yes
1056/826 ... yes
1014/827 ... yes
125/828 ... yes
125/829 ... yes
1215/830 ... yes
105/832 ... yes
105/833 ... yes
1220/834 ... yes
147/835 ... yes
68/836 ... yes
1225/837 ... yes
31/838 ... yes
104/839 ... yes
125/840 ... yes
39/841 ... yes
1056/842 ... yes
353/843 ... yes
1014/844 ... yes
1130/845 ... yes
46/846 ... yes
125/848 ... yes
125/850 ... yes
39/851 ... yes
1056/852 ... yes
296/853 ... yes
1240/854 ... yes
39/855 ... yes
3/856 ... yes
1130/857 ... yes
1056/858 ... yes
226/859 ... yes
11/860 ... yes
274/861 ... yes
1225/862 ... yes
353/863 ... yes
1279/864 ... yes
17/865 ... yes
1198/866 ... yes
125/867 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (3.0.6)
Git user has default SSH configuration? ... yes
Active users: ... 208
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... yes (elasticsearch 7.17.10)
All migrations must be finished before doing a major upgrade ... no (You have 16 pending migrations.)
For more information see:
https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html#all-migrations-must-be-finished-before-doing-a-major-upgrade
Try fixing it:
Wait for all advanced search migrations to complete.
To list pending migrations, run sudo gitlab-rake gitlab:elastic:list_pending_migrations


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

diff --git a/app/services/packages/debian/extract_metadata_service.rb b/app/services/packages/debian/extract_metadata_service.rb
index eb8227d12967..59c274065014 100644
--- a/app/services/packages/debian/extract_metadata_service.rb
+++ b/app/services/packages/debian/extract_metadata_service.rb
@@ -37,7 +37,7 @@ def file_type_basic
 
       def file_type_source
         # https://manpages.debian.org/buster/dpkg-dev/dpkg-source.1.en.html
-        %i[gzip bzip2 lzma xz].each do |format|
+        %i[gzip bzip2 lzma xz zst].each do |format|
           return :source if package_file.file_name.end_with?(".tar.#{format}")
         end
  • add test
  • show better error message when any invalid file is detected
Edited by Philipp Hahn