# Purpose of this epic To plan a path to make the package registry more performant, scalable, and maintainable. This epic collects some ideas that were previously scattered through a few different issues and provides a place to continue looking at the idea of restructuring the data and/or redesigning the schema. This topic is a slow moving piece of technical debt that is not yet of any paramount concern. As package continues to grow and scale in the coming years, more issues may appear, and it will become more difficult to make any major changes like some of the ones proposed below. # Goals - Performance optimization - Improved maintainability - Long term scalability # Background The [package registry](https://docs.gitlab.com/ee/user/packages/package_registry/) manages about a dozen package types. On the ~backend, we have a single table for all packages, and a single table for all package files. These tables map to single models in the code: - `Packages::Package` => `packages_packages` - `Packages::PackageFile` => `packages_package_files` We then have `metadata` tables and models for each individual package type as necessary, for example: - `Packages::MavenMetadatum` => `packages_maven_metadata` As the package registry has grown, we have faced a few challenges: 1. The `Packages::Package` has become difficult to maintain, it contains a large number of methods and validations that are conditional based on the package type. 1. Group-level queries, such as fetching all packages for a group, are underperforming. Instance-level queries often are not usable, especially on GitLab.com due to performance concerns. 1. Although the general idea of how packages work is the same - you are uploading and downloading files - in practice, the different formats have very different [file structures, conventions, validations, and access patterns](https://gitlab.com/gitlab-org/gitlab/-/issues/358205). Trying to put these all into a single model has led to the conditional methods mentioned in (1.). In the past, this subject has come up from time to time: - [Investigating the initial intent of the structure](https://gitlab.com/gitlab-org/gitlab/-/issues/32805) - [Discussion on the idea of splitting](https://gitlab.com/gitlab-org/gitlab/-/issues/199743) the `packages_packages` table - [When looking at how maven packages are structured differently than our modeling](https://gitlab.com/gitlab-org/gitlab/-/issues/299555) - This led to a [thread discussing "what if?"](https://gitlab.com/gitlab-org/gitlab/-/issues/199743#note_479787775) topics on if we could design the schema from scratch. # Potential solutions ## Model Split Using composition or inheritance, each package format has its own class. This seems likely to be needed regardless of if any of the other solutions below are considered. - Level of effort: low - Performance optimization :x: - Improved maintainability :ballot_box_with_check: - Long term scalability :question: This solution is essentially refactoring the model without really changing any underlying logic. ## Indexing We could research access patterns to see if there is room for improvement in how we are querying our data as it currently is stored. - Level of effort: low - Performance optimization :question: - Improved maintainability :x: - Long term scalability :question: ## Table Split (includes model split) Analysis issue: https://gitlab.com/gitlab-org/gitlab/-/issues/356905 - Level of effort: medium-high - Performance optimization :question: - Improved maintainability :ballot_box_with_check: - Long term scalability :ballot_box_with_check: This takes splitting the model one step further to splitting the `packages_packages` table by package type and using [polymorphic associations](https://docs.gitlab.com/ee/development/polymorphic_associations.html). So we would have tables like `maven_packages`, `nuget_packages`, etc... ### Table partition An alternative solution to splitting the table would be to partition by package type. This could potentially help with certain types of queries. ## Table Restructure Analysis issue: https://gitlab.com/gitlab-org/gitlab/-/issues/356905 - Level of effort: high - Performance optimization :question: - Improved maintainability :ballot_box_with_check: - Long term scalability :ballot_box_with_check: This involves updating the design of the tables from: ```mermaid classDiagram Project <|-- Package Package <|-- PackageFile ``` to: ```mermaid classDiagram Project <|-- Package Package <|-- PackageVersion Package <|-- PackageFile PackageVersion <|-- PackageFile ``` Renaming the existing package `Package` as `PackageVersion` and inserting a new `Package` or `PackageEntity` between the `PackageVersion` and `Project`. The perceived benefit of this type of design is we can associate files with the overall package, not just a single version. The maven `metadata.xml` file is a file that would have such an association. This does not necessarily involve splitting the table by package type, but is meant to model the package contents in a way that is closer to what we see on many package formats. ## Table Restructure per package type - Performance optimization :question: - Improved maintainability :question: - Long term scalability :ballot_box_with_check: - Level of effort: very high This is the most far from what we currently have. This would involve taking the table restructuring a step further and structuring per package type. This is not a likely solution, but investigating and discussing it may lead to some interesting ideas and insights we could use in other ways. # Concerns - Carrierwave file association: Uploaded files are stored using their ID or package ID as a location identifier. If we start moving tables, IDs change, and we need to handle moving them in object storage. # Research First we should research whether or not the three goals above are attainable and worth the investment. We can: - Collect and organize data on our package formats: https://gitlab.com/gitlab-org/gitlab/-/issues/358205 - Investigate some of the solutions above in more detail: determine if they are worth pursuing through some analysis, look at the difficulty of implementation if they are worth pursuing, and then looking at the return on investment.