Draft: Debian repository models

What does this MR do?

Debian package models, in preparation of the pakcage upload and repo presentation endpoints.

This is item 2.4 of #5835 (comment 414103286)

Discussion

We need to support different file types (see spec/fixtures/packages/debian of !44029 (merged) for an example):

• .deb files: the most common package files
• .udeb files: lightened .deb files, used for debian-installer
• .tar.{gz,bz2,xz,...}: Source files
• .dsc: Source metadata, and list of source files (with hashes)
• .buildinfo: Used for Reproducible builds
• .changes: Upload metadata, and list of uploaded files (all the above)

Uploading with dput will send those files, one at a time (a different HTTP request for each), finishing by the changes file.

With this MR:

• Package maps to a tuple source_package, source_version
• All file types above can be attached as PackageFile to such a Package by extracting source_package, source_version from the .changes file uploaded later on

Usually, the files are put in a incoming directory, and the changes files are parsed regularly. incoming is mapped as a fake Package, with name=incoming and version=nil.

It should also be possible to upload standalone .debs, mostly from vendor supplied packages. For this, I will later add the ability to skip incoming (and either set source_package, source_version using parameters, or try to guess from the filename).

There is also the concept of distribution which has metadata also (see http://ftp.fr.debian.org/debian/dists/bullseye/Release for the Debian bullseye). Some metadata is generated, but some should be configurable. The current implementation stores those metadata in new tables.

DB Migrations

Up :

== 20201124095300 CreatePackagesDebianProjectDistributions: migrating =========
-- table_exists?(:packages_debian_project_distributions)
   -> 0.0015s
-- create_table(:packages_debian_project_distributions, {})
   -> 0.0887s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_834dabadb6\nCHECK ( char_length(codename) <= 255 )\nNOT VALID;\n")
   -> 0.0008s
-- current_schema()
   -> 0.0005s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_834dabadb6;")
   -> 0.0030s
-- execute("RESET ALL")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_a56ae58a17\nCHECK ( char_length(suite) <= 255 )\nNOT VALID;\n")
   -> 0.0009s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_a56ae58a17;")
   -> 0.0029s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_6177ccd4a6\nCHECK ( char_length(origin) <= 255 )\nNOT VALID;\n")
   -> 0.0012s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_6177ccd4a6;")
   -> 0.0034s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_6f6b55a4c4\nCHECK ( char_length(label) <= 255 )\nNOT VALID;\n")
   -> 0.0009s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_6f6b55a4c4;")
   -> 0.0033s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0008s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_96965792c2\nCHECK ( char_length(version) <= 255 )\nNOT VALID;\n")
   -> 0.0014s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_96965792c2;")
   -> 0.0032s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_b93154339f\nCHECK ( char_length(description) <= 255 )\nNOT VALID;\n")
   -> 0.0012s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_b93154339f;")
   -> 0.0031s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_c25603a25b\nCHECK ( char_length(encrypted_signing_keys) <= 2048 )\nNOT VALID;\n")
   -> 0.0010s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_c25603a25b;")
   -> 0.0026s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_d488f8cce3\nCHECK ( char_length(encrypted_signing_keys_iv) <= 255 )\nNOT VALID;\n")
   -> 0.0008s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_d488f8cce3;")
   -> 0.0027s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_a5d608cb8c\nCHECK ( char_length(release_file) <= 255 )\nNOT VALID;\n")
   -> 0.0009s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_a5d608cb8c;")
   -> 0.0028s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE packages_debian_project_distributions\nADD CONSTRAINT check_adc71cc0de\nCHECK ( char_length(release_file_signature) <= 4096 )\nNOT VALID;\n")
   -> 0.0014s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE packages_debian_project_distributions VALIDATE CONSTRAINT check_adc71cc0de;")
   -> 0.0025s
== 20201124095300 CreatePackagesDebianProjectDistributions: migrated (0.2392s) 

== 20201124100200 CreatePackagesDebianGroupDistributions: migrating ===========
-- table_exists?(:packages_debian_group_distributions)
   -> 0.0006s
-- create_table(:packages_debian_group_distributions, {})
   -> 0.0588s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_590e18405a\nCHECK ( char_length(codename) <= 255 )\nNOT VALID;\n")
   -> 0.0010s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_590e18405a;")
   -> 0.0026s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_e7c928a24b\nCHECK ( char_length(suite) <= 255 )\nNOT VALID;\n")
   -> 0.0007s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_e7c928a24b;")
   -> 0.0027s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0003s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_b057cd840a\nCHECK ( char_length(origin) <= 255 )\nNOT VALID;\n")
   -> 0.0007s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_b057cd840a;")
   -> 0.0028s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_d3244bfc0b\nCHECK ( char_length(label) <= 255 )\nNOT VALID;\n")
   -> 0.0007s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_d3244bfc0b;")
   -> 0.0031s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_3fdadf4a0c\nCHECK ( char_length(version) <= 255 )\nNOT VALID;\n")
   -> 0.0007s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_3fdadf4a0c;")
   -> 0.0030s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_310ac457b8\nCHECK ( char_length(description) <= 255 )\nNOT VALID;\n")
   -> 0.0010s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_310ac457b8;")
   -> 0.0029s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_b811ec1218\nCHECK ( char_length(encrypted_signing_keys) <= 2048 )\nNOT VALID;\n")
   -> 0.0018s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_b811ec1218;")
   -> 0.0028s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0004s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_9b90bc0f07\nCHECK ( char_length(encrypted_signing_keys_iv) <= 255 )\nNOT VALID;\n")
   -> 0.0010s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_9b90bc0f07;")
   -> 0.0030s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_a34c3d776f\nCHECK ( char_length(release_file) <= 255 )\nNOT VALID;\n")
   -> 0.0009s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_a34c3d776f;")
   -> 0.0030s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_group_distributions\nADD CONSTRAINT check_163c88d7bb\nCHECK ( char_length(release_file_signature) <= 4096 )\nNOT VALID;\n")
   -> 0.0009s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_group_distributions VALIDATE CONSTRAINT check_163c88d7bb;")
   -> 0.0028s
== 20201124100200 CreatePackagesDebianGroupDistributions: migrated (0.1963s) ==

== 20201124152800 CreatePackagesDebianProjectArchitectures: migrating =========
-- table_exists?(:packages_debian_project_architectures)
   -> 0.0011s
-- create_table(:packages_debian_project_architectures, {})
   -> 0.0430s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_project_architectures\nADD CONSTRAINT check_9c2e1c99d8\nCHECK ( char_length(name) <= 255 )\nNOT VALID;\n")
   -> 0.0010s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_project_architectures VALIDATE CONSTRAINT check_9c2e1c99d8;")
   -> 0.0033s
== 20201124152800 CreatePackagesDebianProjectArchitectures: migrated (0.0647s) 

== 20201124153000 CreatePackagesDebianGroupArchitectures: migrating ===========
-- table_exists?(:packages_debian_group_architectures)
   -> 0.0012s
-- create_table(:packages_debian_group_architectures, {})
   -> 0.0405s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_group_architectures\nADD CONSTRAINT check_ddb220164a\nCHECK ( char_length(name) <= 255 )\nNOT VALID;\n")
   -> 0.0010s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_group_architectures VALIDATE CONSTRAINT check_ddb220164a;")
   -> 0.0037s
== 20201124153000 CreatePackagesDebianGroupArchitectures: migrated (0.0657s) ==

== 20201124153200 CreatePackagesDebianProjectComponents: migrating ============
-- table_exists?(:packages_debian_project_components)
   -> 0.0012s
-- create_table(:packages_debian_project_components, {})
   -> 0.0448s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_project_components\nADD CONSTRAINT check_517559f298\nCHECK ( char_length(name) <= 255 )\nNOT VALID;\n")
   -> 0.0013s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_project_components VALIDATE CONSTRAINT check_517559f298;")
   -> 0.0033s
== 20201124153200 CreatePackagesDebianProjectComponents: migrated (0.0698s) ===

== 20201124154100 CreatePackagesDebianGroupComponents: migrating ==============
-- table_exists?(:packages_debian_group_components)
   -> 0.0019s
-- create_table(:packages_debian_group_components, {})
   -> 0.0433s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_group_components\nADD CONSTRAINT check_a9bc7d85be\nCHECK ( char_length(name) <= 255 )\nNOT VALID;\n")
   -> 0.0012s
-- current_schema()
   -> 0.0008s
-- execute("ALTER TABLE packages_debian_group_components VALIDATE CONSTRAINT check_a9bc7d85be;")
   -> 0.0030s
== 20201124154100 CreatePackagesDebianGroupComponents: migrated (0.0782s) =====

== 20201124154300 CreatePackagesDebianProjectComponentFiles: migrating ========
-- table_exists?(:packages_debian_project_component_files)
   -> 0.0013s
-- create_table(:packages_debian_project_component_files, {})
   -> 0.0449s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_project_component_files\nADD CONSTRAINT check_e5af03fa2d\nCHECK ( char_length(file) <= 255 )\nNOT VALID;\n")
   -> 0.0008s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_project_component_files VALIDATE CONSTRAINT check_e5af03fa2d;")
   -> 0.0030s
== 20201124154300 CreatePackagesDebianProjectComponentFiles: migrated (0.0660s) 

== 20201124161800 CreatePackagesDebianGroupComponentFiles: migrating ==========
-- table_exists?(:packages_debian_group_component_files)
   -> 0.0011s
-- create_table(:packages_debian_group_component_files, {})
   -> 0.0417s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0008s
-- execute("ALTER TABLE packages_debian_group_component_files\nADD CONSTRAINT check_839e1685bc\nCHECK ( char_length(file) <= 255 )\nNOT VALID;\n")
   -> 0.0013s
-- current_schema()
   -> 0.0006s
-- execute("ALTER TABLE packages_debian_group_component_files VALIDATE CONSTRAINT check_839e1685bc;")
   -> 0.0031s
== 20201124161800 CreatePackagesDebianGroupComponentFiles: migrated (0.0670s) =

== 20201126140700 CreatePackagesDebianPublications: migrating =================
-- create_table(:packages_debian_publications, {})
   -> 0.0355s
== 20201126140700 CreatePackagesDebianPublications: migrated (0.0357s) ========

== 20201126140800 CreatePackagesDebianFileMetadata: migrating =================
-- table_exists?(:packages_debian_file_metadata)
   -> 0.0015s
-- create_table(:packages_debian_file_metadata, {})
   -> 0.0359s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_file_metadata\nADD CONSTRAINT check_2ebedda4b6\nCHECK ( char_length(component) <= 255 )\nNOT VALID;\n")
   -> 0.0015s
-- current_schema()
   -> 0.0008s
-- execute("ALTER TABLE packages_debian_file_metadata VALIDATE CONSTRAINT check_2ebedda4b6;")
   -> 0.0036s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0007s
-- execute("ALTER TABLE packages_debian_file_metadata\nADD CONSTRAINT check_e6e1fffcca\nCHECK ( char_length(architecture) <= 255 )\nNOT VALID;\n")
   -> 0.0012s
-- current_schema()
   -> 0.0005s
-- execute("ALTER TABLE packages_debian_file_metadata VALIDATE CONSTRAINT check_e6e1fffcca;")
   -> 0.0054s
== 20201126140800 CreatePackagesDebianFileMetadata: migrated (0.0743s) ========

Down:

== 20201126140800 CreatePackagesDebianFileMetadata: reverting =================
-- drop_table(:packages_debian_file_metadata)
   -> 0.0675s
== 20201126140800 CreatePackagesDebianFileMetadata: reverted (0.0677s) ========

== 20201126140700 CreatePackagesDebianPublications: reverting =================
-- drop_table(:packages_debian_publications, {})
   -> 0.0085s
== 20201126140700 CreatePackagesDebianPublications: reverted (0.0104s) ========

== 20201124161800 CreatePackagesDebianGroupComponentFiles: reverting ==========
-- drop_table(:packages_debian_group_component_files)
   -> 0.0303s
== 20201124161800 CreatePackagesDebianGroupComponentFiles: reverted (0.0304s) =

== 20201124154300 CreatePackagesDebianProjectComponentFiles: reverting ========
-- drop_table(:packages_debian_project_component_files)
   -> 0.0162s
== 20201124154300 CreatePackagesDebianProjectComponentFiles: reverted (0.0162s) 

== 20201124154100 CreatePackagesDebianGroupComponents: reverting ==============
-- drop_table(:packages_debian_group_components)
   -> 0.0139s
== 20201124154100 CreatePackagesDebianGroupComponents: reverted (0.0140s) =====

== 20201124153200 CreatePackagesDebianProjectComponents: reverting ============
-- drop_table(:packages_debian_project_components)
   -> 0.0101s
== 20201124153200 CreatePackagesDebianProjectComponents: reverted (0.0102s) ===

== 20201124153000 CreatePackagesDebianGroupArchitectures: reverting ===========
-- drop_table(:packages_debian_group_architectures)
   -> 0.0057s
== 20201124153000 CreatePackagesDebianGroupArchitectures: reverted (0.0058s) ==

== 20201124152800 CreatePackagesDebianProjectArchitectures: reverting =========
-- drop_table(:packages_debian_project_architectures)
   -> 0.0081s
== 20201124152800 CreatePackagesDebianProjectArchitectures: reverted (0.0082s) 

== 20201124100200 CreatePackagesDebianGroupDistributions: reverting ===========
-- drop_table(:packages_debian_group_distributions)
   -> 0.0127s
== 20201124100200 CreatePackagesDebianGroupDistributions: reverted (0.0128s) ==

== 20201124095300 CreatePackagesDebianProjectDistributions: reverting =========
-- drop_table(:packages_debian_project_distributions)
   -> 0.0183s
== 20201124095300 CreatePackagesDebianProjectDistributions: reverted (0.0185s) 

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• [-] Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• [-] Label as security and @ mention @gitlab-com/gl-security/appsec
• [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• [-] Security reports checked/validated by a reviewer from the AppSec team