Support maven plugins packaging in the maven metadata sync worker (!56229) · Merge requests · GitLab.org / GitLab

David Fernandez requested to merge 11424-sync-maven-metadata-for-maven-plugins into master Mar 10, 2021

🎎 Context

The Maven package registry is used to host user's maven artifacts.

The Maven protocol uses a maven-metadata.xml as a way to describe the package registry state, among other things it can list the available versions for a given package. This file is totally generated and handled by the clients ($ mvn and $ gradle). The package registry will simply treat it as an additional package file.

Now, when a new maven package is uploaded, two records ::Packages::Package are created:

one with the version to host the package artifacts (usually a .jar file and a .pom file)
one without a version to host the maven-metadata.xml file. This one is also called the versionless package.
- This is expected because maven-metadata.xml file is not tied to any version, it is thus sent without a version.

The problem we're facing is that, a user can remove a package version using the GitLab UI. By doing that, the maven-metadata.xml file can reference non existing versions and guess what happens when mvn or gradle try to pull a non existing version of a package? Yes, that's right 💥 (404)

That's ~bug #11424 (closed).

The above was fixed with !55207 (merged) by having a background worker that is responsible to sync the versions from the database with the ones in the xml file. If a difference is found, a new xml file is generated and saved in the related package record.

Maven artifacts can have different packagings. The usual one is jar but we can also have maven-plugin.

During the analysis of this ~bug, we discovered that the maven-plugin packaging also generates an additional maven-metadata.xml file. This list the available plugins for a given groupId. In other words, maven-plugins have not one but two maven-metadata.xml files. One for the versions and one for the available plugins.

When a new maven-plugin is uploaded, three record ::Packages::Package are created:

one with the package name and version for the artifacts files
one with the package name and no version for the maven-metadata.xml that lists the versions
- Here is an example
one with the groupId and no version for the maven-metadata.xml that lists the plugins
- Here is an example

We do have the same original issue here, which is: if a user deletes the last version of a maven-plugin, the maven-metadata.xml for plugins should be updated so that it doesn't reference a non existing plugin.

🤔 What does this MR do?

Updates the services used by Packages::Maven::Metadata::SyncWorker to support maven-plugins.
- Create a new service to generate the proper xml structure if necessary (Packages::Maven::Metadata::CreatePluginsXmlService)
- Update the orchestrating service(Packages::Maven::Metadata::SyncService) to update plugins
- Re-use the existing service to append a new package file with a given content (Packages::Maven::Metadata::AppendPackageFileService).
Updates the related specs

🖼 Screenshots (strongly suggested)

Setup

We're going to create the following artifacts/versions (using https://gitlab.com/10io/gl_pru which is merely a script that setups a dummy empty package and uploads it to the given url)

one-maven-plugin
- 0.3
- 0.5
- 1.0-SNAPSHOT
- 1.5
- 2.0-SNAPSHOT
- 1.4
two-maven-plugin
- 1.3
maven-package
- 2.3

maven-package is a standard jar artifact. It will be in the same groupId as the plugins. It will help us to verify that changes applied to maven plugin xml only affect maven plugins and not the other maven packages.

one-maven-plugin creation commands

$ bundle exec thor package:push --package-type=maven_plugin --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/73/packages/maven --name=maven.package --version=0.3
[snip ... snip]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.3/maven.package-0.3.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.3/maven.package-0.3.jar (12 kB at 6.2 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.3/maven.package-0.3.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.3/maven.package-0.3.pom (6.2 kB at 3.8 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml (320 B at 138 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml (224 B at 128 B/s)


$ bundle exec thor package:push --package-type=maven_plugin --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/73/packages/maven --name=maven.package --version=0.5
[snip ... snip]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.5/maven.package-0.5.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.5/maven.package-0.5.jar (12 kB at 6.9 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.5/maven.package-0.5.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/0.5/maven.package-0.5.pom (6.2 kB at 3.8 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml (320 B at 602 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml (224 B at 395 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml (349 B at 218 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml (224 B at 135 B/s)


$ bundle exec thor package:push --package-type=maven_plugin --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/73/packages/maven --name=maven.package --version=1.0-SNAPSHOT
[snip ... snip]
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.0-SNAPSHOT/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.0-SNAPSHOT/maven.package-1.0-20210310.150513-1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.0-SNAPSHOT/maven.package-1.0-20210310.150513-1.jar (12 kB at 7.2 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.0-SNAPSHOT/maven.package-1.0-20210310.150513-1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.0-SNAPSHOT/maven.package-1.0-20210310.150513-1.pom (6.2 kB at 3.7 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml (349 B at 593 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml (224 B at 415 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.0-SNAPSHOT/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.0-SNAPSHOT/maven-metadata.xml (763 B at 465 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml (396 B at 245 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml (224 B at 137 B/s)


$ bundle exec thor package:push --package-type=maven_plugin --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/73/packages/maven --name=maven.package --version=1.5
[snip ... snip]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.5/maven.package-1.5.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.5/maven.package-1.5.jar (12 kB at 6.7 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.5/maven.package-1.5.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/1.5/maven.package-1.5.pom (6.2 kB at 3.6 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml (396 B at 681 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml (224 B at 424 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven.package/maven-metadata.xml (416 B at 252 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/73/packages/maven/gl/pru/maven-metadata.xml (224 B at 133 B/s)


$ bundle exec thor package:push --package-type=maven_plugin --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/73/packages/maven --name=maven.package --version=2.0-SNAPSHOT
[snip ... snip]

two-maven-plugin creation commands

$ bundle exec thor package:push --package-type=maven_plugin --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/74/packages/maven --name=two-maven-plugin --version=1.3
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/two-maven-plugin/1.3/two-maven-plugin-1.3.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/two-maven-plugin/1.3/two-maven-plugin-1.3.jar (12 kB at 6.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/two-maven-plugin/1.3/two-maven-plugin-1.3.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/two-maven-plugin/1.3/two-maven-plugin-1.3.pom (6.2 kB at 3.8 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/two-maven-plugin/maven-metadata.xml
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-metadata.xml (225 B at 391 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/two-maven-plugin/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/two-maven-plugin/maven-metadata.xml (323 B at 193 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-metadata.xml (363 B at 211 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.334 s
[INFO] Finished at: 2021-03-10T16:14:09+01:00
[INFO] ------------------------------------------------------------------------

maven-package creation commands

$ bundle exec thor package:push --package-type=maven --user=root --token=XXX --url=http://gdk.test:8000/api/v4/projects/74/packages/maven --name=maven-package --version=1.2
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-package/1.2/maven-package-1.2.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-package/1.2/maven-package-1.2.jar (2.2 kB at 1.2 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-package/1.2/maven-package-1.2.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-package/1.2/maven-package-1.2.pom (1.3 kB at 764 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-package/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/74/packages/maven/gl/pru/maven-package/maven-metadata.xml (295 B at 172 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.880 s
[INFO] Finished at: 2021-03-10T16:15:16+01:00
[INFO] ------------------------------------------------------------------------

Let's look at the UI:

All the artifacts are there, nothing to say here.

Let's look at the xmls:

`one-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>one-maven-plugin</artifactId>
  <versioning>
    <latest>1.4</latest>
    <release>1.4</release>
    <versions>
      <version>0.3</version>
      <version>0.5</version>
      <version>1.0-SNAPSHOT</version>
      <version>1.5</version>
      <version>2.0-SNAPSHOT</version>
      <version>1.4</version>
    </versions>
    <lastUpdated>20210310151156</lastUpdated>
  </versioning>
</metadata>

`two-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>two-maven-plugin</artifactId>
  <versioning>
    <latest>1.3</latest>
    <release>1.3</release>
    <versions>
      <version>1.3</version>
    </versions>
    <lastUpdated>20210310151406</lastUpdated>
  </versioning>
</metadata>

plugins xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <plugins>
    <plugin>
      <name>one-maven-plugin</name>
      <prefix>one</prefix>
      <artifactId>one-maven-plugin</artifactId>
    </plugin>
    <plugin>
      <name>two-maven-plugin</name>
      <prefix>two</prefix>
      <artifactId>two-maven-plugin</artifactId>
    </plugin>
  </plugins>
</metadata>

Delete versions

Delete `one-maven-plugin` `1.4`

`one-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>one-maven-plugin</artifactId>
  <versioning>
    <latest>2.0-SNAPSHOT</latest>
    <release>1.5</release>
    <versions>
      <version>0.3</version>
      <version>0.5</version>
      <version>1.0-SNAPSHOT</version>
      <version>1.5</version>
      <version>2.0-SNAPSHOT</version>
    </versions>
    <lastUpdated>20210310152343</lastUpdated>
  </versioning>
</metadata>

`one-maven-plugin` versions xml

```xml gl.pru two-maven-plugin 1.3 1.3 1.3 20210310151406 ```

plugins xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <plugins>
    <plugin>
      <name>one-maven-plugin</name>
      <prefix>one</prefix>
      <artifactId>one-maven-plugin</artifactId>
    </plugin>
    <plugin>
      <name>two-maven-plugin</name>
      <prefix>two</prefix>
      <artifactId>two-maven-plugin</artifactId>
    </plugin>
  </plugins>
</metadata>

versions xml for one-maven-plugin updated accordingly
versions xml for two-maven-plugin not changed
plugins xml not changed

Delete `one-maven-plugin` `2.0-SNAPSHOT`

`one-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>one-maven-plugin</artifactId>
  <versioning>
    <latest>1.5</latest>
    <release>1.5</release>
    <versions>
      <version>0.3</version>
      <version>0.5</version>
      <version>1.0-SNAPSHOT</version>
      <version>1.5</version>
    </versions>
    <lastUpdated>20210310152809</lastUpdated>
  </versioning>
</metadata>

`two-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>two-maven-plugin</artifactId>
  <versioning>
    <latest>1.3</latest>
    <release>1.3</release>
    <versions>
      <version>1.3</version>
    </versions>
    <lastUpdated>20210310151406</lastUpdated>
  </versioning>
</metadata>

plugins versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <plugins>
    <plugin>
      <name>one-maven-plugin</name>
      <prefix>one</prefix>
      <artifactId>one-maven-plugin</artifactId>
    </plugin>
    <plugin>
      <name>two-maven-plugin</name>
      <prefix>two</prefix>
      <artifactId>two-maven-plugin</artifactId>
    </plugin>
  </plugins>
</metadata>

versions xml for one-maven-plugin updated accordingly
versions xml for two-maven-plugin not changed
plugins xml not changed

Delete `one-maven-plugin` `1.5`

`one-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>one-maven-plugin</artifactId>
  <versioning>
    <latest>1.0-SNAPSHOT</latest>
    <release>0.5</release>
    <versions>
      <version>0.3</version>
      <version>0.5</version>
      <version>1.0-SNAPSHOT</version>
    </versions>
    <lastUpdated>20210310154614</lastUpdated>
  </versioning>
</metadata>

`two-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>two-maven-plugin</artifactId>
  <versioning>
    <latest>1.3</latest>
    <release>1.3</release>
    <versions>
      <version>1.3</version>
    </versions>
    <lastUpdated>20210310151406</lastUpdated>
  </versioning>
</metadata>

plugins versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <plugins>
    <plugin>
      <name>one-maven-plugin</name>
      <prefix>one</prefix>
      <artifactId>one-maven-plugin</artifactId>
    </plugin>
    <plugin>
      <name>two-maven-plugin</name>
      <prefix>two</prefix>
      <artifactId>two-maven-plugin</artifactId>
    </plugin>
  </plugins>
</metadata>

versions xml for one-maven-plugin updated accordingly
versions xml for two-maven-plugin not changed
plugins xml not changed

Delete `one-maven-plugin` `0.5`

`one-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>one-maven-plugin</artifactId>
  <versioning>
    <latest>1.0-SNAPSHOT</latest>
    <release>0.3</release>
    <versions>
      <version>0.3</version>
      <version>1.0-SNAPSHOT</version>
    </versions>
    <lastUpdated>20210310160003</lastUpdated>
  </versioning>
</metadata>

`two-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>two-maven-plugin</artifactId>
  <versioning>
    <latest>1.3</latest>
    <release>1.3</release>
    <versions>
      <version>1.3</version>
    </versions>
    <lastUpdated>20210310151406</lastUpdated>
  </versioning>
</metadata>

plugins versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <plugins>
    <plugin>
      <name>one-maven-plugin</name>
      <prefix>one</prefix>
      <artifactId>one-maven-plugin</artifactId>
    </plugin>
    <plugin>
      <name>two-maven-plugin</name>
      <prefix>two</prefix>
      <artifactId>two-maven-plugin</artifactId>
    </plugin>
  </plugins>
</metadata>

versions xml for one-maven-plugin updated accordingly
versions xml for two-maven-plugin not changed
plugins xml not changed

Delete `one-maven-plugin` `0.3`

`one-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>one-maven-plugin</artifactId>
  <versioning>
    <latest>1.0-SNAPSHOT</latest>
    <versions>
      <version>1.0-SNAPSHOT</version>
    </versions>
    <lastUpdated>20210310161537</lastUpdated>
  </versioning>
</metadata>

`two-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>two-maven-plugin</artifactId>
  <versioning>
    <latest>1.3</latest>
    <release>1.3</release>
    <versions>
      <version>1.3</version>
    </versions>
    <lastUpdated>20210310151406</lastUpdated>
  </versioning>
</metadata>

plugins versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <plugins>
    <plugin>
      <name>one-maven-plugin</name>
      <prefix>one</prefix>
      <artifactId>one-maven-plugin</artifactId>
    </plugin>
    <plugin>
      <name>two-maven-plugin</name>
      <prefix>two</prefix>
      <artifactId>two-maven-plugin</artifactId>
    </plugin>
  </plugins>
</metadata>

versions xml for one-maven-plugin updated accordingly
versions xml for two-maven-plugin not changed
plugins xml not changed

Delete `one-maven-plugin` `1.0-SNAPSHOT`

The interesting action 😺 because it's removing the last version of one-maven-plugin

`one-maven-plugin` versions xml

Versionless package for versions xml destroyed along with the related package files. This one-maven-plugin versions xml is thus no more available (404)

`two-maven-plugin` versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>gl.pru</groupId>
  <artifactId>two-maven-plugin</artifactId>
  <versioning>
    <latest>1.3</latest>
    <release>1.3</release>
    <versions>
      <version>1.3</version>
    </versions>
    <lastUpdated>20210310151406</lastUpdated>
  </versioning>
</metadata>

plugins versions xml

<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <plugins>
    <plugin>
      <name>two-maven-plugin</name>
      <prefix>two</prefix>
      <artifactId>two-maven-plugin</artifactId>
    </plugin>
  </plugins>
</metadata>

versions xml for one-maven-plugin destroyed along with the related versionless package 🎉
versions xml for two-maven-plugin not changed
plugins xml updated accordingly
- only two-maven-plugin is listed 🎉

Delete `two-maven-plugin` `1.3`

Another interesting action: we're removing the very last version of the very last plugin of a given groupId. Two versionless packages should be removed.

`two-maven-plugin` versions xml

Versionless package for versions xml destroyed along with the related package files. This two-maven-plugin versions xml is thus no more available (404)

plugins versions xml

Plugins xml destroyed along with the related package files. This file is thus no more available (404)

versions xml for two-maven-plugin destroyed 🎉
plugins xml destroyed 🎉

Final view of packages from a rails console:

[1] pry(main)> Project.find(74).packages.maven
=> [#<Packages::Package:0x00007fad5e8dc9e0 id: 308, project_id: 74, created_at: Wed, 10 Mar 2021 15:15:15 UTC +00:00, updated_at: Wed, 10 Mar 2021 15:15:15 UTC +00:00, name: "gl/pru/maven-package", version: nil, package_type: "maven", creator_id: 1, status: "default">,
 #<Packages::Package:0x00007fad5e9ee950 id: 307, project_id: 74, created_at: Wed, 10 Mar 2021 15:15:11 UTC +00:00, updated_at: Wed, 10 Mar 2021 15:15:11 UTC +00:00, name: "gl/pru/maven-package", version: "1.2", package_type: "maven", creator_id: 1, status: "default">]

Only the jar maven package remains with its versionless package. 🎉

📏 Does this MR meet the acceptance criteria?

Conformity

📋 Does this MR need a changelog?
- I have included a changelog entry.
- I have not included a changelog entry because _____.
[-] Documentation (if required)
Code review guidelines
Merge request performance guidelines
Style guides
Database guides
[-] Separation of EE specific content

⚙ Availability and Testing

[-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
[-] Tested in all supported browsers
[-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

🛃 Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

[-] Label as security and @ mention @gitlab-com/gl-security/appsec
[-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
[-] Security reports checked/validated by a reviewer from the AppSec team

💾 Database review

!56229 (comment 527232815)

Query:

SELECT "packages_maven_metadata"."app_name"
  FROM "packages_maven_metadata"
 WHERE "packages_maven_metadata"."package_id" IN (
        SELECT "packages_packages"."id"
          FROM "packages_packages"
         WHERE "packages_packages"."project_id" = 17012483
           AND "packages_packages"."package_type" = 1
           AND "packages_packages"."status" = 0
           AND "packages_packages"."name" IN ('gl/pru/my-maven-plugin', 'gl/pru/my-second-maven-plugin', 'gl/pru/plugin-1-maven-plugin', 'gl/pru/plugin-2-maven-plugin', 'gl/pru/plugin-3-maven-plugin', 'gl/pru/plugin-4-maven-plugin', 'gl/pru/plugin-5-maven-plugin', 'gl/pru/plugin-6-maven-plugin', 'gl/pru/plugin-7-maven-plugin', 'gl/pru/plugin-8-maven-plugin', 'gl/pru/plugin-9-maven-plugin', 'gl/pru/plugin-10-maven-plugin', 'gl/pru/plugin-11-maven-plugin', 'gl/pru/plugin-12-maven-plugin', 'gl/pru/plugin-13-maven-plugin', 'gl/pru/plugin-14-maven-plugin', 'gl/pru/plugin-15-maven-plugin', 'gl/pru/plugin-16-maven-plugin', 'gl/pru/plugin-17-maven-plugin', 'gl/pru/plugin-18-maven-plugin', 'gl/pru/plugin-19-maven-plugin', 'gl/pru/plugin-20-maven-plugin', 'gl/pru/plugin-21-maven-plugin', 'gl/pru/plugin-22-maven-plugin', 'gl/pru/plugin-23-maven-plugin', 'gl/pru/plugin-24-maven-plugin', 'gl/pru/plugin-25-maven-plugin', 'gl/pru/plugin-26-maven-plugin', 'gl/pru/plugin-27-maven-plugin', 'gl/pru/plugin-28-maven-plugin', 'gl/pru/plugin-29-maven-plugin', 'gl/pru/plugin-30-maven-plugin', 'gl/pru/plugin-31-maven-plugin', 'gl/pru/plugin-32-maven-plugin', 'gl/pru/plugin-33-maven-plugin', 'gl/pru/plugin-34-maven-plugin', 'gl/pru/plugin-35-maven-plugin', 'gl/pru/plugin-36-maven-plugin', 'gl/pru/plugin-37-maven-plugin', 'gl/pru/plugin-38-maven-plugin', 'gl/pru/plugin-39-maven-plugin', 'gl/pru/plugin-40-maven-plugin', 'gl/pru/plugin-41-maven-plugin', 'gl/pru/plugin-42-maven-plugin', 'gl/pru/plugin-43-maven-plugin', 'gl/pru/plugin-44-maven-plugin', 'gl/pru/plugin-45-maven-plugin', 'gl/pru/plugin-46-maven-plugin', 'gl/pru/plugin-47-maven-plugin', 'gl/pru/plugin-48-maven-plugin', 'gl/pru/plugin-49-maven-plugin', 'gl/pru/plugin-50-maven-plugin', 'gl/pru/plugin-51-maven-plugin', 'gl/pru/plugin-52-maven-plugin', 'gl/pru/plugin-53-maven-plugin', 'gl/pru/plugin-54-maven-plugin', 'gl/pru/plugin-55-maven-plugin', 'gl/pru/plugin-56-maven-plugin', 'gl/pru/plugin-57-maven-plugin', 'gl/pru/plugin-58-maven-plugin', 'gl/pru/plugin-59-maven-plugin', 'gl/pru/plugin-60-maven-plugin', 'gl/pru/plugin-61-maven-plugin', 'gl/pru/plugin-62-maven-plugin', 'gl/pru/plugin-63-maven-plugin', 'gl/pru/plugin-64-maven-plugin', 'gl/pru/plugin-65-maven-plugin', 'gl/pru/plugin-66-maven-plugin', 'gl/pru/plugin-67-maven-plugin', 'gl/pru/plugin-68-maven-plugin', 'gl/pru/plugin-69-maven-plugin', 'gl/pru/plugin-70-maven-plugin', 'gl/pru/plugin-71-maven-plugin', 'gl/pru/plugin-72-maven-plugin', 'gl/pru/plugin-73-maven-plugin', 'gl/pru/plugin-74-maven-plugin', 'gl/pru/plugin-75-maven-plugin', 'gl/pru/plugin-76-maven-plugin', 'gl/pru/plugin-77-maven-plugin', 'gl/pru/plugin-78-maven-plugin', 'gl/pru/plugin-79-maven-plugin', 'gl/pru/plugin-80-maven-plugin', 'gl/pru/plugin-81-maven-plugin', 'gl/pru/plugin-82-maven-plugin', 'gl/pru/plugin-83-maven-plugin', 'gl/pru/plugin-84-maven-plugin', 'gl/pru/plugin-85-maven-plugin', 'gl/pru/plugin-86-maven-plugin', 'gl/pru/plugin-87-maven-plugin', 'gl/pru/plugin-88-maven-plugin', 'gl/pru/plugin-89-maven-plugin', 'gl/pru/plugin-90-maven-plugin', 'gl/pru/plugin-91-maven-plugin', 'gl/pru/plugin-92-maven-plugin', 'gl/pru/plugin-93-maven-plugin', 'gl/pru/plugin-94-maven-plugin', 'gl/pru/plugin-95-maven-plugin', 'gl/pru/plugin-96-maven-plugin', 'gl/pru/plugin-97-maven-plugin', 'gl/pru/plugin-98-maven-plugin', 'gl/pru/plugin-99-maven-plugin', 'gl/pru/plugin-100-maven-plugin')
           AND "packages_packages"."version" IS NOT NULL
       )
 ORDER BY created_at ASC

Explain plan:

Sort  (cost=215.73..215.73 rows=1 width=28) (actual time=192.531..192.544 rows=140 loops=1)
   Sort Key: packages_maven_metadata.created_at
   Sort Method: quicksort  Memory: 35kB
   Buffers: shared hit=1035 read=224 dirtied=18
   I/O Timings: read=179.761
   ->  Nested Loop  (cost=1.10..215.72 rows=1 width=28) (actual time=15.831..192.195 rows=140 loops=1)
         Buffers: shared hit=1032 read=224 dirtied=18
         I/O Timings: read=179.761
         ->  Index Scan using idx_packages_packages_on_project_id_name_version_package_type on public.packages_packages  (cost=0.55..212.14 rows=1 width=8) (actual time=7.687..88.315 rows=140 loops=1)
               Index Cond: ((packages_packages.project_id = 17012483) AND ((packages_packages.name)::text = ANY ('{gl/pru/my-maven-plugin,gl/pru/my-second-maven-plugin,gl/pru/plugin-1-maven-plugin,gl/pru/plugin-2-maven-plugin,gl/pru/plugin-3-maven-plugin,gl/pru/plugin-4-maven-plugin,gl/pru/plugin-5-maven-plugin,gl/pru/plugin-6-maven-plugin,gl/pru/plugin-7-maven-plugin,gl/pru/plugin-8-maven-plugin,gl/pru/plugin-9-maven-plugin,gl/pru/plugin-10-maven-plugin,gl/pru/plugin-11-maven-plugin,gl/pru/plugin-12-maven-plugin,gl/pru/plugin-13-maven-plugin,gl/pru/plugin-14-maven-plugin,gl/pru/plugin-15-maven-plugin,gl/pru/plugin-16-maven-plugin,gl/pru/plugin-17-maven-plugin,gl/pru/plugin-18-maven-plugin,gl/pru/plugin-19-maven-plugin,gl/pru/plugin-20-maven-plugin,gl/pru/plugin-21-maven-plugin,gl/pru/plugin-22-maven-plugin,gl/pru/plugin-23-maven-plugin,gl/pru/plugin-24-maven-plugin,gl/pru/plugin-25-maven-plugin,gl/pru/plugin-26-maven-plugin,gl/pru/plugin-27-maven-plugin,gl/pru/plugin-28-maven-plugin,gl/pru/plugin-29-maven-plugin,gl/pru/plugin-30-maven-plugin,gl/pru/plugin-31-maven-plugin,gl/pru/plugin-32-maven-plugin,gl/pru/plugin-33-maven-plugin,gl/pru/plugin-34-maven-plugin,gl/pru/plugin-35-maven-plugin,gl/pru/plugin-36-maven-plugin,gl/pru/plugin-37-maven-plugin,gl/pru/plugin-38-maven-plugin,gl/pru/plugin-39-maven-plugin,gl/pru/plugin-40-maven-plugin,gl/pru/plugin-41-maven-plugin,gl/pru/plugin-42-maven-plugin,gl/pru/plugin-43-maven-plugin,gl/pru/plugin-44-maven-plugin,gl/pru/plugin-45-maven-plugin,gl/pru/plugin-46-maven-plugin,gl/pru/plugin-47-maven-plugin,gl/pru/plugin-48-maven-plugin,gl/pru/plugin-49-maven-plugin,gl/pru/plugin-50-maven-plugin,gl/pru/plugin-51-maven-plugin,gl/pru/plugin-52-maven-plugin,gl/pru/plugin-53-maven-plugin,gl/pru/plugin-54-maven-plugin,gl/pru/plugin-55-maven-plugin,gl/pru/plugin-56-maven-plugin,gl/pru/plugin-57-maven-plugin,gl/pru/plugin-58-maven-plugin,gl/pru/plugin-59-maven-plugin,gl/pru/plugin-60-maven-plugin,gl/pru/plugin-61-maven-plugin,gl/pru/plugin-62-maven-plugin,gl/pru/plugin-63-maven-plugin,gl/pru/plugin-64-maven-plugin,gl/pru/plugin-65-maven-plugin,gl/pru/plugin-66-maven-plugin,gl/pru/plugin-67-maven-plugin,gl/pru/plugin-68-maven-plugin,gl/pru/plugin-69-maven-plugin,gl/pru/plugin-70-maven-plugin,gl/pru/plugin-71-maven-plugin,gl/pru/plugin-72-maven-plugin,gl/pru/plugin-73-maven-plugin,gl/pru/plugin-74-maven-plugin,gl/pru/plugin-75-maven-plugin,gl/pru/plugin-76-maven-plugin,gl/pru/plugin-77-maven-plugin,gl/pru/plugin-78-maven-plugin,gl/pru/plugin-79-maven-plugin,gl/pru/plugin-80-maven-plugin,gl/pru/plugin-81-maven-plugin,gl/pru/plugin-82-maven-plugin,gl/pru/plugin-83-maven-plugin,gl/pru/plugin-84-maven-plugin,gl/pru/plugin-85-maven-plugin,gl/pru/plugin-86-maven-plugin,gl/pru/plugin-87-maven-plugin,gl/pru/plugin-88-maven-plugin,gl/pru/plugin-89-maven-plugin,gl/pru/plugin-90-maven-plugin,gl/pru/plugin-91-maven-plugin,gl/pru/plugin-92-maven-plugin,gl/pru/plugin-93-maven-plugin,gl/pru/plugin-94-maven-plugin,gl/pru/plugin-95-maven-plugin,gl/pru/plugin-96-maven-plugin,gl/pru/plugin-97-maven-plugin,gl/pru/plugin-98-maven-plugin,gl/pru/plugin-99-maven-plugin,gl/pru/plugin-100-maven-plugin}'::text[])) AND (packages_packages.version IS NOT NULL) AND (packages_packages.package_type = 1))
               Filter: (packages_packages.status = 0)
               Rows Removed by Filter: 0
               Buffers: shared hit=452 read=100 dirtied=6
               I/O Timings: read=85.417
         ->  Index Scan using index_packages_maven_metadata_on_package_id_and_path on public.packages_maven_metadata  (cost=0.55..3.57 rows=1 width=36) (actual time=0.732..0.738 rows=1 loops=140)
               Index Cond: (packages_maven_metadata.package_id = packages_packages.id)
               Buffers: shared hit=580 read=124 dirtied=12
               I/O Timings: read=94.343

https://console.postgres.ai/shared/323d6003-c992-4252-9dee-466689716410

Edited Mar 15, 2021 by David Fernandez

Support maven plugins packaging in the maven metadata sync worker

🎎 Context

🤔 What does this MR do?

🖼 Screenshots (strongly suggested)

Setup

Delete versions

Delete one-maven-plugin 1.4

Delete one-maven-plugin 2.0-SNAPSHOT

Delete one-maven-plugin 1.5

Delete one-maven-plugin 0.5

Delete one-maven-plugin 0.3

Delete one-maven-plugin 1.0-SNAPSHOT

Delete two-maven-plugin 1.3

Final view of packages from a rails console:

📏 Does this MR meet the acceptance criteria?

Conformity

⚙ Availability and Testing

🛃 Security

💾 Database review

!56229 (comment 527232815)

Merge request reports

Delete `one-maven-plugin` `1.4`

Delete `one-maven-plugin` `2.0-SNAPSHOT`

Delete `one-maven-plugin` `1.5`

Delete `one-maven-plugin` `0.5`

Delete `one-maven-plugin` `0.3`

Delete `one-maven-plugin` `1.0-SNAPSHOT`

Delete `two-maven-plugin` `1.3`