Skip to content

Maven virtual registries: introduce remove local upstreams worker

David Fernandezrequested to merge
566217-delete-local-upstream-when-target-is-destroyed into master

⚖️ Context

In Maven virtual registries local upstream support... (!206725 - merged) • David Fernandez • 18.6, we introduced a local mode for the maven upstreams. In local mode, the ::VirtualRegistries::Packages::Maven::Upstream will reference a Project or a Group by storing their global_id in the url field. For more details on this part, see !206725 (merged).

Now, when a Project or a Group is destroyed, we need to remove those local upstreams. When it comes to upstreams, we can't simply delete the database record and that's it. Upstreams are organized as an ordered list in a ::VirtualRegistries::Packages::Maven::Registry. Keeping the list in a coherent order (or better said positioning) is essential for the core logic of virtual registries. Thus, when an upstream is destroyed, we need to update the positions of all siblings upstreams that are "above" the destroyed upstream.

In other words, we can't simply cascade the deletes in the database, we also need to execute some logic in the rails side. To fulfil this need, we're going to use the existing delete (project/group) events of the EventStore. This will allow us to implement a callback that will look for any existing upstream and destroy it. While destroying it, we will also resync the positions of all remaining siblings upstreams.

🤔 What does this MR do and why?

  • Create a DestroyLocalUpstream worker that will listen to ::Projects::ProjectDeletedEvent and ::Groups::GroupDeletedEvent.
  • Add the related specs.

Due to https://docs.gitlab.com/development/event_store/#register-the-subscriber-to-the-event, the event store subscription will be added in a follow up MR: Maven virtual registries: add event store subsc... (!211752) • David Fernandez • 18.6.

📚 References

🌈 Screenshots or screen recordings

No UI changes

⚙️ How to set up and validate locally

Requirements:

  • Have an EE license as virtual registries are EE only feature.

(Make sure that the rails-background-jobs gdk process is running as the EventStore callback is baked by Sidekiq.)

There are no APIs or UIs for this feature (yet), so we are going to test it in a Rails console:

project_to_delete = FactoryBot.create(:project) # if the path is already taken, just re-run the command until it succeeds.
top_level_group = Group.all.select(&:root?).sample
upstream = ::VirtualRegistries::Packages::Maven::Upstream.create!(name: 'testing', url: project_to_delete.to_global_id.to_s, group: top_level_group)

current_user = User.first # this user should have the rights to destroy the project. In gdk, this one is root.

# trigger the job
::VirtualRegistries::DestroyLocalUpstreamsWorker.new.perform('Projects::ProjectDeletedEvent', ::Projects::ProjectDeletedEvent.new(data: { project_id: project_to_delete.id, namespace_id: project_to_delete.namespace_id }).data)

# check that the upstream are deleted
upstream.reload
# ActiveRecord::RecordNotFound: Couldn't find VirtualRegistries::Packages::Maven::Upstream with [WHERE "virtual_registries_packages_maven_upstreams"."id" = $1]

You can try the exact same scenario but with a group_to_delete = FactoryBot.create(:group). It will work the same way

🏎️ MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

✅

💾 Database analysis

⬆️ Migration up

$ rails db:migrate:up:main VERSION=20251030132327 && r db:migrate:up:ci VERSION=20251030132327
main: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 79279
main: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: migrating 
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0343s
main: -- index_exists?(:virtual_registries_packages_maven_upstreams, :url, {:name=>"index_virtual_registries_packages_maven_upstreams_on_url", :algorithm=>:concurrently})
main:    -> 0.0016s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- add_index(:virtual_registries_packages_maven_upstreams, :url, {:name=>"index_virtual_registries_packages_maven_upstreams_on_url", :algorithm=>:concurrently})
main:    -> 0.0026s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: migrated (0.0749s) 

main: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 79279
ci: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 79471
ci: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: migrating 
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0353s
ci: -- index_exists?(:virtual_registries_packages_maven_upstreams, :url, {:name=>"index_virtual_registries_packages_maven_upstreams_on_url", :algorithm=>:concurrently})
ci:    -> 0.0019s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0004s
ci: -- add_index(:virtual_registries_packages_maven_upstreams, :url, {:name=>"index_virtual_registries_packages_maven_upstreams_on_url", :algorithm=>:concurrently})
ci:    -> 0.0039s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0003s
ci: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: migrated (0.0874s) 

ci: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 79471

⬇️ Migration down

$ rails db:migrate:down:main VERSION=20251030132327 && r db:migrate:down:ci VERSION=20251030132327
main: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 78890
main: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: reverting 
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0348s
main: -- indexes(:virtual_registries_packages_maven_upstreams)
main:    -> 0.0019s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- remove_index(:virtual_registries_packages_maven_upstreams, {:algorithm=>:concurrently, :name=>"index_virtual_registries_packages_maven_upstreams_on_url"})
main:    -> 0.0022s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: reverted (0.0750s) 

main: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 78890
ci: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 79062
ci: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: reverting 
ci: -- transaction_open?(nil)
ci:    -> 0.0000s
ci: -- view_exists?(:postgres_partitions)
ci:    -> 0.0376s
ci: -- indexes(:virtual_registries_packages_maven_upstreams)
ci:    -> 0.0026s
ci: -- execute("SET statement_timeout TO 0")
ci:    -> 0.0005s
ci: -- remove_index(:virtual_registries_packages_maven_upstreams, {:algorithm=>:concurrently, :name=>"index_virtual_registries_packages_maven_upstreams_on_url"})
ci:    -> 0.0023s
ci: -- execute("RESET statement_timeout")
ci:    -> 0.0004s
ci: == 20251030132327 AddIndexOnUrlOnVirtualRegistriesPackagesMavenUpstreams: reverted (0.0889s) 

ci: == [advisory_lock_connection] object_id: 130140, pg_backend_pid: 79062

⚙️ Queries

Edited by David Fernandez

Merge request reports