Draft: Maven Virtual Registry Upstream model changes for local mode

🧦 Context

In the Maven virtual registry world, we pull files from an upstream through the GitLab virtual registry. While doing so, we also cache the requested file so that subsequent requests are served solely by GitLab and not by the upstream.

The modelization is as follows:

Registry <-n:1- RegistryUpstream -1:n-> Upstream <-n:1- CacheEntry

Basically, a Registry can have multiple Upstreams (through a join model RegistryUpstream) and an Upstream has many cache entries.

Up to now, an Upstream was an url with optional credentials. That would define how to access a remote upstream.

With Maven virtual registries: local upstreams (#548558) • Bonnie Tsang • 18.11 • At risk, we want to introduce the concept of a local upstream. Instead of looking for files in remote upstreams, we look at the GitLab Maven package registry. In short words, we point to a target project or target group and we inspect the (Maven) packages available at that project or group. To handle this, we need to update the existing logic to inspect local upstreams. Before that, we need to define what we need for a local upstream.

See more details in the approach in !206725 (merged).

🗒️ Implementation plan

This change being quite deep in the existing logic, we're going to split it in multiple MRs:

1. Upstream (pure) database changes. !206725 (merged).
2. Upstream mode changes. 👈 This is this MR.
3. CacheEntry model and database changes.
4. Update the services layer logic.
5. Manage the local target destruction logic. eg. what happens when a project or group targeted by a local upstream is destroyed.
6. Update the APIs logic. This is the client that manages upstreams (CRUD operations).
7. Update the documentation.

🤔 What does this MR do and why?

• Update the model validations by categorizing them: some will run when remote and others will run when remote.
• Update the related specs.

📖 References

• Maven virtual registries: local upstreams backe... (#566217 - closed) • David Fernandez • 18.9 • At risk.
• Maven virtual registries: local upstreams (#548558) • Bonnie Tsang • 18.11 • At risk.
• Maven Virtual Registry - Road to General Availa... (&15089) • Tim Rizzi, Crystal Poole+ • 19.2 • On track.

🖥️ Screenshots or screen recordings

No UI changes

⚗️ How to set up and validate locally

Since we don't have any client that can create or update local upstream objects yet (that's the existing Rest API), we're going to use a rails console:

top_level_group = Group.top_level.find_by_path('gitlab-org')
project = top_level_group.projects.first

registry = ::VirtualRegistries::Packages::Maven::Registry.create!(group: top_level_group, name: 'test local upstreams')

# invalid remote registry
upstream1 = ::VirtualRegistries::Packages::Maven::Upstream.create!(group: registry.group, url: 'https://repo1.maven.org', name: 'invalid remote upstream', mode: :local)
ActiveRecord::RecordInvalid: Validation failed: Target group can't be blank, Target project can't be blank, Url must be blank

# valid remote registry
upstream1 = ::VirtualRegistries::Packages::Maven::Upstream.create!(group: registry.group, url: 'https://repo1.maven.org', name: 'remote upstream', mode: :remote)

# invalid local registry
upstream2 = ::VirtualRegistries::Packages::Maven::Upstream.create!(group: registry.group, username: 'test', target_project_id: project.id, name: 'invalid local project upstream', mode: :local)
ActiveRecord::RecordInvalid: Validation failed: Username must be blank

# valid local registry
upstream2 = ::VirtualRegistries::Packages::Maven::Upstream.create!(group: registry.group, target_project_id: project.id, name: 'invalid local project upstream', mode: :local)

# invalid local registry pointing to a group and a project
upstream3 = ::VirtualRegistries::Packages::Maven::Upstream.create!(group: registry.group, target_project_id: Project.last.id, target_group_id: registry.group.id, name: 'invalid local project upstream', mode: :local)
ActiveRecord::RecordInvalid: Validation failed: target_project_id and target_group_id are mutually exclusive. Set only one of these.

upstream3 = ::VirtualRegistries::Packages::Maven::Upstream.create!(group: registry.group, target_group_id: registry.group.id, name: 'invalid local project upstream', mode: :local)

# Now, let's demonstrate that we can't add local project upstream if a parent is already part of the existing local upstreams

::VirtualRegistries::Packages::Maven::RegistryUpstream.create!(group: registry.group, registry: registry, upstream: upstream3) # adding the top level group as the first local upstream Ok.
::VirtualRegistries::Packages::Maven::RegistryUpstream.create!(group: registry.group, registry: registry, upstream: upstream1) # adding the remote upstream as the second one. Ok.
::VirtualRegistries::Packages::Maven::RegistryUpstream.create!(group: registry.group, registry: registry, upstream: upstream2) # adding the (sub) project as the 3rd local upstream fails because the parent group is already included (first upstream in the list).

🏁 MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

💽 Database analysis

⚙️ Queries

TBD