Sync the maven metadata file upon package deletion (!55207) · Merge requests · GitLab.org / GitLab

David Fernandez requested to merge 11424-background-worker-for-updating-the-maven-metadata-xml-file into master Feb 25, 2021

🍩 Context

The `maven-metadata.xml` file

When maven clients (such as mvn or gradle) interact with the GitLab Maven Packages registry, they will expect and interact with a special file: the maven-metadata.xml file.

The aim of this file is to have a "manifest" of all the available versions for a given package.

This file is fully generated and updated by the clients meaning that when you push a brand new maven package, this file will be created and uploaded. When you push a new version to an existing maven package, the existing file will be downloaded, updated and re-uploaded.

Now, when clients upload this file, there is no version attached to it and that it's logical because this file is not linked to a particular version. It's a manifest for a given maven package name but not a given version. That is why when clients upload a brand new package, two ::Packages::Package records will be created:

one with the version that will host the maven archive itself
one without the version (also called the "versionless" package) that will host solely the maven-metadata.xml

Please note that we're not talking about the special maven-metadata.xml file that is generated for snapshot versions.

Lastly, depending on the maven package "packaging type" one or two maven-metadata.xml are used. More in depth analysis here. This MR deals only with the metadata file for the most common packaging: jar.

⚠ Handling the second maven-metadata.xml file for maven-plugin packages will be done in a follow up MR. ⚠

🔬 A note on `release` and `latest` fields

mvn and gradle will fill these fields by looking at the packages ordered by their creation timestamp and not by their version string.

Here is what mvn and gradle will generate as latest / release.

With mvn

(mvn doesn't generate the <latest> element)

Uploading version	release	latest
1.3	1.3
2.0-SNAPSHOT	1.3
1.6	1.6
1.4	1.4
1.5-SNAPSHOT	1.4

With gradle

Uploading version	release	latest
1.3	1.3	1.3
2.0-SNAPSHOT	1.3	2.0-SNAPSHOT
1.6	1.6	1.6
1.4	1.4	1.4
1.5-SNAPSHOT	1.4	1.5-SNAPSHOT

🔥 The problem

Now, imagine what happens when a user uses the GitLab UI to delete a maven package? Yes, that's right 💥 . The maven-metadata.xml is not updated and will become unsynced with the existing versions from the database. In other words, the maven-metadata.xml might reference a version that doesn't exist anymore and this will be a bigger issue when other maven applications will pull this package: they can try to pull a version that doesn't exist 💥

That is exactly issue #11424 (closed).

How do we sync back, the versions in the maven-metadata.xml file and those in the database?

We're going to use the simple and boring approach here: using a background worker. Basically, each time a maven package is destroyed, a job for this worker is enqueued.

🚒 What does this MR do?

(For simplicity and to ease the maintainering of this code, this code has been centralized in a brand new namespace: Packages::Maven::Metadata)

Adds a new worker to sync the maven-metadata.xml file
The worker will use 3 different services to accomplish that:
- Packages::Maven::Metadata::SyncService
  - Orchestrates the whole thing.
  - In charge of locating the maven-metadata.xml file and reading it.
- Packages::Maven::Metadata::CreateVersionsXmlService
  - In charge of receiving the current content of the maven-metadata.xml and producing an updated one if necessary using the versions that are in the database.
  - Basically, encapsulates all the Nokogiri usage for this change.
- Packages::Maven::Metadata::AppendPackageFileService
  - Merely a "util" service to create (append) a package file to a given package.
  - Since it's not a straightforward Packages::PackageFile record, this service will handle all the logic around that model.
Adds/Updates the related specs
Updates the packages rest api delete endpoint to enqueue a job when a maven package is destroyed.

📸 Screenshots (strongly suggested)

The following examples make heavy use of https://gitlab.com/10io/gl_pru which is a simple script that creates a dummy package of the given type and uploads it to the given registry url.

With `mvn`

Setup

Let's publish versions 0.7, 1.0-SNAPSHOT, 1.0, 1.1 and 2.0-SNAPSHOT of the same package respecting the order:

Package versions creation

$ bundle exec thor package:push --package-type=maven --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyMavenPackage --version=0.7
[...snip..]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/0.7/MyDummyMavenPackage-0.7.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/0.7/MyDummyMavenPackage-0.7.jar (2.3 kB at 1.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/0.7/MyDummyMavenPackage-0.7.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/0.7/MyDummyMavenPackage-0.7.pom (1.3 kB at 791 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (301 B at 178 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.667 s
[INFO] Finished at: 2021-03-02T10:38:08+01:00
[INFO] ------------------------------------------------------------------------

$ bundle exec thor package:push --package-type=maven --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyMavenPackage --version=1.0-SNAPSHOT
[...snip..]
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0-SNAPSHOT/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0-SNAPSHOT/MyDummyMavenPackage-1.0-20210302.093901-1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0-SNAPSHOT/MyDummyMavenPackage-1.0-20210302.093901-1.jar (2.3 kB at 1.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0-SNAPSHOT/MyDummyMavenPackage-1.0-20210302.093901-1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0-SNAPSHOT/MyDummyMavenPackage-1.0-20210302.093901-1.pom (1.3 kB at 786 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (301 B at 585 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0-SNAPSHOT/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0-SNAPSHOT/maven-metadata.xml (769 B at 479 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (339 B at 205 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.746 s
[INFO] Finished at: 2021-03-02T10:39:08+01:00
[INFO] ------------------------------------------------------------------------

$ bundle exec thor package:push --package-type=maven --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyMavenPackage --version=1.0
[...snip..]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0/MyDummyMavenPackage-1.0.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0/MyDummyMavenPackage-1.0.jar (2.3 kB at 1.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0/MyDummyMavenPackage-1.0.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.0/MyDummyMavenPackage-1.0.pom (1.3 kB at 772 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (339 B at 650 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (368 B at 228 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.989 s
[INFO] Finished at: 2021-03-02T10:39:35+01:00
[INFO] ------------------------------------------------------------------------

$ bundle exec thor package:push --package-type=maven --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyMavenPackage --version=1.1
[...snip..]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.1/MyDummyMavenPackage-1.1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.1/MyDummyMavenPackage-1.1.jar (2.3 kB at 1.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.1/MyDummyMavenPackage-1.1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/1.1/MyDummyMavenPackage-1.1.pom (1.3 kB at 763 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (368 B at 680 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (397 B at 246 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.893 s
[INFO] Finished at: 2021-03-02T10:40:35+01:00
[INFO] ------------------------------------------------------------------------

$ bundle exec thor package:push --package-type=maven --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyMavenPackage --version=2.0-SNAPSHOT
[...snip..]
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/2.0-SNAPSHOT/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/2.0-SNAPSHOT/MyDummyMavenPackage-2.0-20210302.094104-1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/2.0-SNAPSHOT/MyDummyMavenPackage-2.0-20210302.094104-1.jar (2.3 kB at 1.4 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/2.0-SNAPSHOT/MyDummyMavenPackage-2.0-20210302.094104-1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/2.0-SNAPSHOT/MyDummyMavenPackage-2.0-20210302.094104-1.pom (1.3 kB at 803 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (397 B at 763 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/2.0-SNAPSHOT/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/2.0-SNAPSHOT/maven-metadata.xml (769 B at 452 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyMavenPackage/maven-metadata.xml (435 B at 250 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.841 s
[INFO] Finished at: 2021-03-02T10:41:11+01:00
[INFO] ------------------------------------------------------------------------

Here is what we have in the UI:

Here is the maven-metadata.xml content:

XML content

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>MyDummyMavenPackage</artifactId>
  <versioning>
    <release>1.1</release>
    <versions>
      <version>0.7</version>
      <version>1.0-SNAPSHOT</version>
      <version>1.0</version>
      <version>1.1</version>
      <version>2.0-SNAPSHOT</version>
    </versions>
    <lastUpdated>20210302094108</lastUpdated>
  </versioning>
</metadata>

A few things to notice with mvn:

There is no <latest> element
<release> points to the most recently non snapshot version created

Delete versions from the UI

Let's see how the metadata content is updated when deleting package versions.

Delete version 2.0-SNAPSHOT in the UI

XML content

   <?xml version="1.0" encoding="UTF-8"?>
   <metadata>
     <groupId>gl.pru</groupId>
     <artifactId>MyDummyMavenPackage</artifactId>
     <versioning>
       <release>1.1</release>
       <versions>
         <version>0.7</version>
         <version>1.0-SNAPSHOT</version>
         <version>1.0</version>
         <version>1.1</version>
       </versions>
       <lastUpdated>20210302094912</lastUpdated>
     </versioning>
   </metadata>

The release didn't move
The versions list was updated

Delete version 1.1 in the UI

XML content

   <?xml version="1.0" encoding="UTF-8"?>
   <metadata>
     <groupId>gl.pru</groupId>
     <artifactId>MyDummyMavenPackage</artifactId>
     <versioning>
       <release>1.0</release>
       <versions>
         <version>0.7</version>
         <version>1.0-SNAPSHOT</version>
         <version>1.0</version>
       </versions>
       <lastUpdated>20210302095349</lastUpdated>
     </versioning>
   </metadata>

The release was updated
The versions list was updated

Delete version 1.0 in the UI

XML content

  <?xml version="1.0" encoding="UTF-8"?>
  <metadata>
    <groupId>gl.pru</groupId>
    <artifactId>MyDummyMavenPackage</artifactId>
    <versioning>
      <release>0.7</release>
      <versions>
        <version>0.7</version>
        <version>1.0-SNAPSHOT</version>
      </versions>
      <lastUpdated>20210302102653</lastUpdated>
    </versioning>
  </metadata>

The release was updated
The versions list was updated

Delete version 0.7 in the UI

XML content

  <?xml version="1.0" encoding="UTF-8"?>
  <metadata>
    <groupId>gl.pru</groupId>
    <artifactId>MyDummyMavenPackage</artifactId>
    <versioning>
      <versions>
        <version>1.0-SNAPSHOT</version>
      </versions>
      <lastUpdated>20210302102942</lastUpdated>
    </versioning>
  </metadata>

The release was removed because the only version left is a snapshot one
The versions list was updated

Delete version 1.0-SNAPSHOT in the UI
This being the last version, removing it will trigger a destruction of the versionless package. From the worker logs (the extra.packages_maven_metadata_sync_worker.message key has the message Versionless package for versions destroyed)

{"severity":"INFO","time":"2021-03-02T10:59:21.694Z","class":"Packages::Maven::Metadata::SyncWorker","args":["1","66","gl/pru/MyDummyMavenPackage"],"retry":3,"queue":"package_repositories:packages_maven_metadata_sync","backtrace":true,"version":0,"queue_namespace":"package_repositories","jid":"96c3eede22c9232177dc622d","created_at":"2021-03-02T10:59:20.904Z","meta.user":"root","meta.project":"root/sgddsgf","meta.root_namespace":"root","meta.caller_id":"/api/:version/projects/:id/packages/:package_id","meta.remote_ip":"127.0.0.1","meta.feature_category":"package_registry","correlation_id":"01EZS9F919GDD0XSBG3XYP4R2D","enqueued_at":"2021-03-02T10:59:20.905Z","pid":80101,"message":"Packages::Maven::Metadata::SyncWorker JID-96c3eede22c9232177dc622d: done: 0.78693 sec","job_status":"done","scheduling_latency_s":0.002113,"job_size_bytes":591,"cpu_s":0.316603,"db_count":12,"db_write_count":3,"db_cached_count":0,"extra.packages_maven_metadata_sync_worker.message":"Versionless package for versions destroyed","duration_s":0.78693,"completed_at":"2021-03-02T10:59:21.694Z","db_duration_s":0.075225}

With `gradle`

Setup

Let's publish versions 0.7, 1.0-SNAPSHOT, 1.0, 1.1 and 2.0-SNAPSHOT of the same package respecting the order:

Package versions creation

$ bundle exec thor package:push --package-type=gradle --user=root --token=XXXX --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyGradlePackage --version=0.7 

$ bundle exec thor package:push --package-type=gradle --user=root --token=cLvDvyom1yy5BjhjeMQx --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyGradlePackage --version=1.0-SNAPSHOT

$ bundle exec thor package:push --package-type=gradle --user=root --token=cLvDvyom1yy5BjhjeMQx --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyGradlePackage --version=1.0

$ bundle exec thor package:push --package-type=gradle --user=root --token=cLvDvyom1yy5BjhjeMQx --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyGradlePackage --version=1.1

$ bundle exec thor package:push --package-type=gradle --user=root --token=cLvDvyom1yy5BjhjeMQx --url=http://gdk.test:8000/api/v4/projects/66/packages/maven --name=MyDummyGradlePackage --version=2.0-SNAPSHOT

Here is what we have in the UI:

Here is the maven-metadata.xml content:

XML content

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>MyDummyGradlePackage</artifactId>
  <versioning>
    <latest>2.0-SNAPSHOT</latest>
    <release>1.1</release>
    <versions>
      <version>0.7</version>
      <version>1.0-SNAPSHOT</version>
      <version>1.0</version>
      <version>1.1</version>
      <version>2.0-SNAPSHOT</version>
    </versions>
    <lastUpdated>20210302122246</lastUpdated>
  </versioning>
</metadata>

A few things to notice with gradle:

we have a <latest> element. It points to the latest version (snapshot or not)

Delete versions from the UI

Let's see how the metadata content is updated when deleting package versions.

Delete version 2.0-SNAPSHOT in the UI

XML content

  <?xml version="1.0" encoding="UTF-8"?>
  <metadata>
    <groupId>gl.pru</groupId>
    <artifactId>MyDummyGradlePackage</artifactId>
    <versioning>
      <latest>1.1</latest>
      <release>1.1</release>
      <versions>
        <version>0.7</version>
        <version>1.0-SNAPSHOT</version>
        <version>1.0</version>
        <version>1.1</version>
      </versions>
      <lastUpdated>20210302122744</lastUpdated>
    </versioning>
  </metadata>

The latest was updated
The release was updated
The versions list was updated

Delete version 1.0 in the UI

XML content

  <?xml version="1.0" encoding="UTF-8"?>
  <metadata>
    <groupId>gl.pru</groupId>
    <artifactId>MyDummyGradlePackage</artifactId>
    <versioning>
      <latest>1.1</latest>
      <release>1.1</release>
      <versions>
        <version>0.7</version>
        <version>1.0-SNAPSHOT</version>
        <version>1.1</version>
      </versions>
      <lastUpdated>20210302123101</lastUpdated>
    </versioning>
  </metadata>

The latest was not updated
The release was not updated
The versions list was updated

Delete version 1.1 in the UI

XML content

  <?xml version="1.0" encoding="UTF-8"?>
  <metadata>
    <groupId>gl.pru</groupId>
    <artifactId>MyDummyGradlePackage</artifactId>
    <versioning>
      <latest>1.0-SNAPSHOT</latest>
      <release>0.7</release>
      <versions>
        <version>0.7</version>
        <version>1.0-SNAPSHOT</version>
      </versions>
      <lastUpdated>20210302123219</lastUpdated>
    </versioning>
  </metadata>

The latest was updated to the snapshot version
The release was updated to the non snapshot version
The versions list was updated

Delete version 1.0-SNAPSHOT in the UI

XML content

  <?xml version="1.0" encoding="UTF-8"?>
  <metadata>
    <groupId>gl.pru</groupId>
    <artifactId>MyDummyGradlePackage</artifactId>
    <versioning>
      <latest>0.7</latest>
      <release>0.7</release>
      <versions>
        <version>0.7</version>
      </versions>
      <lastUpdated>20210302123340</lastUpdated>
    </versioning>
  </metadata>

The latest was updated
The release was updated
The versions list was updated

Delete version 0.7 in the UI
This being the last version, removing it will trigger a destruction of the versionless package. From the worker logs (the extra.packages_maven_metadata_sync_worker.message key has the message Versionless package for versions destroyed)

{"severity":"INFO","time":"2021-03-02T12:35:17.015Z","class":"Packages::Maven::Metadata::SyncWorker","args":["1","66","gl/pru/MyDummyGradlePackage"],"retry":3,"queue":"package_repositories:packages_maven_metadata_sync","backtrace":true,"version":0,"queue_namespace":"package_repositories","jid":"2aa97607c43d15b7595668de","created_at":"2021-03-02T12:35:16.546Z","meta.user":"root","meta.project":"root/sgddsgf","meta.root_namespace":"root","meta.caller_id":"/api/:version/projects/:id/packages/:package_id","meta.remote_ip":"127.0.0.1","meta.feature_category":"package_registry","correlation_id":"01EZSEYXXK275CHX9CZCA9S8J8","enqueued_at":"2021-03-02T12:35:16.549Z","pid":80101,"message":"Packages::Maven::Metadata::SyncWorker JID-2aa97607c43d15b7595668de: done: 0.464801 sec","job_status":"done","scheduling_latency_s":0.000939,"job_size_bytes":592,"cpu_s":0.168167,"db_count":162,"db_write_count":142,"db_cached_count":0,"extra.packages_maven_metadata_sync_worker.message":"Versionless package for versions destroyed","duration_s":0.464801,"completed_at":"2021-03-02T12:35:17.015Z","db_duration_s":0.036985}

Pulling the release version

Imagine having a package with versions 1.0, 1.1 and 2.0-SNAPSHOT.

With mvn, let's pull the release version:

$ mvn dependency:get -Dartifact=gl.pru:MyDummyGradlePackage:RELEASE -DremoteRepositories=gitlab-maven::::http://gdk.test:8000/api/v4/projects/66/packages/maven
[...snip...]
[INFO] Resolving gl.pru:MyDummyGradlePackage:jar:RELEASE with transitive dependencies
Downloading from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/maven-metadata.xml
Downloaded from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/maven-metadata.xml (394 B at 651 B/s)
Downloading from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/1.1/MyDummyGradlePackage-1.1.pom
Downloaded from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/1.1/MyDummyGradlePackage-1.1.pom

The release version is resolved to 1.1.

Now, we delete version 1.1, clear the mvn cache and pull the release version again:

$ mvn dependency:get -Dartifact=gl.pru:MyDummyGradlePackage:RELEASE -DremoteRepositories=gitlab-maven::::http://gdk.test:8000/api/v4/projects/66/packages/maven
[...snip...]
[INFO] Resolving gl.pru:MyDummyGradlePackage:jar:RELEASE with transitive dependencies
Downloading from central: https://repo.maven.apache.org/maven2/gl/pru/MyDummyGradlePackage/maven-metadata.xml
Downloading from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/maven-metadata.xml
Downloaded from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/maven-metadata.xml (374 B at 659 B/s)
Downloading from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/1.0/MyDummyGradlePackage-1.0.pom
Downloaded from gitlab-maven: http://gdk.test:8000/api/v4/projects/66/packages/maven/gl/pru/MyDummyGradlePackage/1.0/MyDummyGradlePackage-1.0.pom

The release version is resolved to 1.0 🎉

Now, let's try with gradle and the same scenario. Let's pull the release version:

$ gradle build
$ gradle -q dependencies --configuration testRuntimeClasspath
[...snip...]
testRuntimeClasspath - Runtime classpath of source set 'test'.
\--- gl.pru:MyDummyGradlePackage:1.+ -> 1.1
     \--- com.google.guava:guava:28.1-jre
          +--- com.google.guava:failureaccess:1.0.1
          +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
          +--- com.google.code.findbugs:jsr305:3.0.2
          +--- org.checkerframework:checker-qual:2.8.1
          +--- com.google.errorprone:error_prone_annotations:2.3.2
          +--- com.google.j2objc:j2objc-annotations:1.3
          \--- org.codehaus.mojo:animal-sniffer-annotations:1.18

The release version is resolved to 1.1.

We remove now 1.1 from the UI, clear the gradle cache and run the commands again:

$ gradle build
$ gradle -q dependencies --configuration testRuntimeClasspath
[...snip...]
testRuntimeClasspath - Runtime classpath of source set 'test'.
\--- gl.pru:MyDummyGradlePackage:1.+ -> 1.0
     \--- com.google.guava:guava:28.1-jre
          +--- com.google.guava:failureaccess:1.0.1
          +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
          +--- com.google.code.findbugs:jsr305:3.0.2
          +--- org.checkerframework:checker-qual:2.8.1
          +--- com.google.errorprone:error_prone_annotations:2.3.2
          +--- com.google.j2objc:j2objc-annotations:1.3
          \--- org.codehaus.mojo:animal-sniffer-annotations:1.18

The release version is resolved to 1.0. 🎉

Conclusions

All the version removals led to the proper updates of the maven-metadata.xml file! 🎉

In addition, when pulling the latest versions, both clients (mvn and gradle) were able to use the updated maven-metadata.xml file to pull the correct version.

Does this MR meet the acceptance criteria?

⚙ Conformity

⛓ Availability and Testing

[-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
[-] Tested in all supported browsers
[-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

🔒 Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

[-] Label as security and @ mention @gitlab-com/gl-security/appsec
[-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
[-] Security reports checked/validated by a reviewer from the AppSec team

💿 Database

!55207 (comment 520354501)

 Limit  (cost=11.60..11.60 rows=1 width=829) (actual time=0.705..0.707 rows=1 loops=1)
   Buffers: shared hit=6 read=6
   I/O Timings: read=0.633
   ->  Sort  (cost=11.60..11.62 rows=9 width=829) (actual time=0.704..0.705 rows=1 loops=1)
         Sort Key: packages_package_files.id DESC
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=6 read=6
         I/O Timings: read=0.633
         ->  Index Scan using index_packages_package_files_on_package_id_and_file_name on public.packages_package_files  (cost=0.56..11.55 rows=9 width=829) (actual time=0.650..0.683 rows=2 loops=1)
               Index Cond: ((packages_package_files.package_id = 979858) AND ((packages_package_files.file_name)::text = 'maven-metadata.xml'::text))
               Buffers: shared hit=3 read=6
               I/O Timings: read=0.633

!55207 (comment 520355729)

Limit  (cost=3.46..3.47 rows=1 width=84) (actual time=35.839..35.841 rows=1 loops=1)
   Buffers: shared hit=4 read=25
   I/O Timings: read=35.524
   ->  Sort  (cost=3.46..3.47 rows=1 width=84) (actual time=35.837..35.838 rows=1 loops=1)
         Sort Key: packages_packages.id
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=4 read=25
         I/O Timings: read=35.524
         ->  Index Scan using index_packages_packages_on_project_id_and_version on public.packages_packages  (cost=0.42..3.45 rows=1 width=84) (actual time=17.483..35.808 rows=1 loops=1)
               Index Cond: ((packages_packages.project_id = 17012483) AND (packages_packages.version IS NULL))
               Filter: ((packages_packages.package_type = 1) AND (packages_packages.status = 0) AND ((packages_packages.name)::text = 'my/group/id/Asoka'::text))
               Rows Removed by Filter: 23
               Buffers: shared hit=1 read=25
               I/O Timings: read=35.524

!55207 (comment 522218916)

Sort  (cost=3.59..3.59 rows=1 width=22) (actual time=10.278..10.279 rows=2 loops=1)
   Sort Key: packages_packages.created_at
   Sort Method: quicksort  Memory: 25kB
   Buffers: shared hit=3 read=6
   I/O Timings: read=10.154
   ->  Index Scan using idx_packages_packages_on_project_id_name_version_package_type on public.packages_packages  (cost=0.55..3.58 rows=1 width=22) (actual time=9.429..10.258 rows=2 loops=1)
         Index Cond: ((packages_packages.project_id = 17012483) AND ((packages_packages.name)::text = 'my/group/id/Asoka'::text) AND (packages_packages.version IS NOT NULL) AND (packages_packages.package_type = 1))
         Filter: (packages_packages.status = 0)
         Rows Removed by Filter: 0
         Buffers: shared read=6
         I/O Timings: read=10.154

Other considerations

Follow up issue for possible better indexes: #323904

Edited Mar 09, 2021 by David Fernandez

Sync the maven metadata file upon package deletion

🍩 Context

The maven-metadata.xml file

🔬 A note on release and latest fields

🔥 The problem

🚒 What does this MR do?

📸 Screenshots (strongly suggested)

With mvn

Setup

Delete versions from the UI

With gradle

Setup

Delete versions from the UI

Pulling the release version

Conclusions

Does this MR meet the acceptance criteria?

⚙ Conformity

⛓ Availability and Testing

🔒 Security

💿 Database

!55207 (comment 520354501)

!55207 (comment 520355729)

!55207 (comment 522218916)

Other considerations

Merge request reports

The `maven-metadata.xml` file

🔬 A note on `release` and `latest` fields

With `mvn`

With `gradle`