Skip to content

Support Maven artifact classifiers when checking for duplicates

David Fernandez requested to merge 325749-allow-jar-clasifiers-in-maven-duplicates into master

Context

When a package is uploaded to the GitLab Maven package registry, the clients will not send a single file but a set of files.

As such, the backend receives not one but multiple uploads.

The best way to understand this is to present the files (that are sent) in a tree-like view (simplified):

.
└── my-awesome-company
    └── foobar-test-bananas
        ├── 4.3
        │   ├── foobar-test-bananas-4.3.jar
        │   └── foobar-test-bananas-4.3.pom
        └── maven-metadata.xml

In reality, there a bit more files but for the sake of simplicity, we will keep it like this.

Now, maven packages also use what they call SNAPSHOTS versions. They are unreleased, ongoing versions. The uploads of those versions have some extra files:

.
└── my-awesome-company
    └── foobar-test-bananas
        ├── 4.2-SNAPSHOT
        │   ├── foobar-test-bananas-4.2-20230306.162821-1.jar
        │   ├── foobar-test-bananas-4.2-20230306.162821-1.pom
        │   └── maven-metadata.xml
        └── maven-metadata.xml

Notice that the file names contain a dynamic part which is made up from a timestamp (20230306.162821) and an increment (1). This is to make sure that the filenames are really unique. This dynamic part is the same for all the files of the same set.

If we upload a new iteration to version 4.2-SNAPSHOT, we will have the following result:

.
└── my-awesome-company
    └── foobar-test-bananas
        ├── 4.2-SNAPSHOT
        │   ├── foobar-test-bananas-4.2-20230306.162821-1.jar
        │   ├── foobar-test-bananas-4.2-20230306.162821-1.pom
        │   ├── foobar-test-bananas-4.2-20230307.113057-2.jar
        │   ├── foobar-test-bananas-4.2-20230307.113057-2.pom
        │   └── maven-metadata.xml
        └── maven-metadata.xml
  • The dynamic part changed. It's a different timestamp (20230307.113057) and the increment is the next one available (2)

On the other hand, the GitLab Maven package registry provides a feature simply called "duplicates". It allows users to decide if uploading files to an existing package is allowed or not.

This feature currently works by reading the file extension if the file that is being uploaded and checking the existing files. If the file extension already exists and duplicates are not allowed, the upload is not accepted.

🐛 The bug

The problem with the above (maven uploads and duplicates) is that users can choose to upload additional jar files with a different "classifier". Example:

.
└── my-awesome-company
    └── foobar-test-bananas
        ├── 4.3
        │   ├── foobar-test-bananas-4.3.jar
        │   ├── foobar-test-bananas-4.3-javadoc.jar
        │   ├── foobar-test-bananas-4.3-source.jar
        │   └── foobar-test-bananas-4.3.pom
        └── maven-metadata.xml

Notice the additional jars with -javadoc and -source classifiers. Guess what happens when duplicates are not allowed? Yes, 💥 That's because the all three files have the same file extension: .jar so the backend will consider the foobar-test-bananas-4.3-javadoc.jar file as a duplicate of foobar-test-bananas-4.3.jar but in reality, it's not the case.

This is exactly the typebug described in #325749 (closed).

🚑 Fixing the bug

Looking at the example above, it seems that this is trivial to solve: Oh yeah, just compare the full filename (foobar-test-bananas-4.3.jar) with the existing ones and that's it.

Unfortunately, that's half of the solution due to the SNAPSHOT versions. Remember the dynamic part? That part will make all filenames unique = we can't compare the full filename.

In other words, we need to divide the duplicates detection logic in two:

  • For non SNAPSHOT versions: compare the uploaded filename with the existing ones.
  • For SNAPSHOT versions: strip the dynamic part from the filename, do the same for the existing ones and then compare them.

🤔 What does this MR do and why?

  1. Update the spec/services/packages/maven/find_or_create_package_service_spec.rb so that artifact classifiers are supported during the duplicates check.
  2. Update the related specs (including the requests specs).

📺 Screenshots or screen recordings

I'm going to use a project in a group where duplicates are not allowed. Here is the pom.xml that I'm going to use:

pom.xml 
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>my.awesome.company</groupId>
  <artifactId>my-package</artifactId>
  <packaging>jar</packaging>
  <version>2.0</version>
  <name>my-package</name>
  <url>http://maven.apache.org</url>
  <properties>
    <maven.compiler.source>7</maven.compiler.source>
    <maven.compiler.target>7</maven.compiler.target>
  </properties>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <repositories>
    <repository>
      <id>gl_pru</id>
      <url>http://gdk.test:8000/api/v4/projects/23/packages/maven</url>
    </repository>
  </repositories>
  <distributionManagement>
    <repository>
      <id>gl_pru</id>
      <url>http://gdk.test:8000/api/v4/projects/23/packages/maven</url>
    </repository>
    <snapshotRepository>
      <id>gl_pru</id>
      <url>http://gdk.test:8000/api/v4/projects/23/packages/maven</url>
    </snapshotRepository>
  </distributionManagement>

  <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-javadoc-plugin</artifactId>
                <version>2.10.3</version>
                <executions>
                    <execution>
                        <id>attach-javadocs</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Notice that the -javadocs jar is going to be generated and uploaded.

For each use case below, I'm going to upload the same file twice.

1️⃣ With a non SNAPSHOT version

💥 using master 

$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.pom (1.9 kB at 6.5 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1.jar (2.4 kB at 14 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.1/my-package-2.1-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.248 s
[INFO] Finished at: 2023-03-07T10:17:27+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:jar:javadoc:2.1 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]

Upload rejected 💥

That's because the -javadoc.jar file is detected as a duplicate and since duplicates are not allowed, the backend rejects the upload.

🚑 using this MR

$ mvn deploy -s settings.xml
[snip ... snip]
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.pom (1.9 kB at 311 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.jar (2.4 kB at 15 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0-javadoc.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0-javadoc.jar (95 kB at 558 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml (304 B at 746 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.231 s
[INFO] Finished at: 2023-03-07T10:10:45+01:00
[INFO] ------------------------------------------------------------------------

Upload is a success. ✅ See how the -javadoc.jar file was accepted too.

Let's upload a second time:

$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.pom
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0.jar
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/2.0/my-package-2.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.821 s
[INFO] Finished at: 2023-03-07T10:12:08+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:pom:2.0 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]

Upload is rejected because it's a duplicate and we don't allow duplicates ✅

1️⃣ With a SNAPSHOT version

💥 using master 

$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.pom (1.9 kB at 7.2 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1.jar (2.4 kB at 15 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.1-SNAPSHOT/my-package-3.1-20230307.091859-1-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.010 s
[INFO] Finished at: 2023-03-07T10:19:01+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:jar:javadoc:3.1-20230307.091859-1 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]

Upload rejected 💥

Same 🐛 here: the -javadoc.jar file is detected as a duplicate.

🚑 using this MR

$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.pom (1.9 kB at 11 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1.jar (2.4 kB at 10.0 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1-javadoc.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091410-1-javadoc.jar (95 kB at 540 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml (304 B at 3.5 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml (984 B at 6.6 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/maven-metadata.xml (342 B at 2.1 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.530 s
[INFO] Finished at: 2023-03-07T10:14:12+01:00
[INFO] ------------------------------------------------------------------------

Upload accepted ✅

No issues with the -javadoc.jar file.

Let's upload again:

$ mvn deploy -s settings.xml
[snip ... snip]
[INFO] --- deploy:3.0.0:deploy (default-deploy) @ my-package ---
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/maven-metadata.xml (984 B at 9.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091524-2.pom
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091524-2.jar
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/23/packages/maven/my/awesome/company/my-package/3.0-SNAPSHOT/my-package-3.0-20230307.091524-2-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.867 s
[INFO] Finished at: 2023-03-07T10:15:26+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:3.0.0:deploy (default-deploy) on project my-package: Failed to deploy artifacts: Could not transfer artifact my.awesome.company:my-package:pom:3.0-20230307.091524-2 from/to gl_pru (http://gdk.test:8000/api/v4/projects/23/packages/maven): status code: 400, reason phrase: Bad Request (400) -> [Help 1]

Upload not accepted as it is detected as a duplicated ✅

⚙ How to set up and validate locally

You can now try the scenarios above.

🛵 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Fernandez

Merge request reports