Skip to content

Force primary in the maven package registry upload endpoint

David Fernandez requested to merge 362668-maven-upload-endpoint-force-primary-reads into master

🔭 Context

The Maven package registry allows user to store their maven packages within GitLab.

Now, publishing a maven package is not as simple as uploading a single file to GitLab.

Instead a set of files are uploaded:

Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/maven-metadata.xml (756 B at 89 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/foobar-5.2-20230317.130326-2.pom
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/foobar-5.2-20230317.130326-2.pom (1.3 kB at 84 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/foobar-5.2-20230317.130326-2.jar
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/foobar-5.2-20230317.130326-2.jar (2.1 kB at 138 B/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml (384 B at 4.3 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/5.2-SNAPSHOT/maven-metadata.xml (756 B at 49 B/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml (422 B at 27 B/s)

Now, those uploads don't happen in parallel. They happen sequentially.

In GitLab, upon the very first upload request, we will create the package object that stores, among other things, the package name and package version.

As such, all uploads go through this service that the logic can be summarized as:

  1. Locate the package.
  2. Create a package if it doesn't exist.

From our observations, it seems that we have an issue between (1.) and (2.). Basically, (1.) being a read, we could use the replica database but then in (2.) the primary is used.

If we throw in replica lag in that chain of uploads, what happens? Well, we have higher chances to have the following situation for uploads after the first one:

  1. The package is located in the replica and not found.
  2. Not being found, the backend will try to create a package in the primary but 💥 a package already exists there. (see https://gitlab.com/gitlab-org/gitlab/-/blob/64155898e65f1648e36a6719e3b4d05d3fa39770/app/models/packages/package.rb#L68-73).

💡 Solution

The ideal solution would be to use upsert so that we can merge the read + create (if necessary) in a single database call. The problem is that we need a UNIQUE constraint for columns used in the upsert. Unfortunately, we already have maven package duplicates in the related table. So, we will need to first patch the issue, then handle the duplicates (probably, delete the oldest package as it is never returned by the package registry) and finally add the UNIQUE constraint.

In this MR, we use a different approach. We basically force the service to use the primary database. This way, the read + create operations happen both on the primary. As such, they both get a consistent "state" of the database (whether if the package exists or not).

The Maven Repository being one of the top most used packages registry on gitlab.com, we are gating these changes behind a feature flag: packages_registry_maven_uploads_force_primary. The rollout issue is: #397028 (closed).

🐦 Three birds one stone

In %15.11, we are working on 3 different issues that all impact the upload endpoint:

  1. Maven sha1 file race condition (404 error) (#362665 - closed), the upload endpoint returns a 404 and we knew that the replica lag was the root cause.
  2. Maven package creation race condition (400 error) (#362668 - closed), the upload endpoint returns a 400. I investigated this one and found out that the replica lag was also the root cause. That was the primary goal of this fix.
  3. Maven package registry returning 409 when uploa... (#367356 - closed), the upload endpoint returns a 429. No ideas on the root cause. While reproduce the issue (2.) locally with a replica lag of 1min, I stumbled upon 429 responses. This confirms that a replica lag is also a cause for this issue.

In other words, this MR will fix 3 Maven Repository issues.

🔬 What does this MR do and why?

  • Update app/services/packages/maven/find_or_create_package_service.rb to use the primary database all the times.
  • Update the related spec.

📺 Screenshots or screen recordings

None

⚙ How to set up and validate locally

  1. Set up a database replica with some lag following https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/database_load_balancing.md.
  2. Using try to upload a maven package to a project with a PAT.

💥 With the feature flag disabled

The uploads have high chances to fail with a 400, 404 or 409 response.

Example:

$ bundle exec thor package:push --package-type=maven --user=root --token="<pat>" --url="http://gdk.test:8000/api/v4/projects/22/packages/maven" --name="foobar" --version="8.4"
[... snip ...]

Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/8.4/foobar-8.4.pom
[WARNING] Failed to upload checksum to gl/pru/foobar/8.4/foobar-8.4.pom.sha1
org.apache.http.client.HttpResponseException: status code: 404, reason phrase: Not Found (404)
    at org.eclipse.aether.transport.http.HttpTransporter.handleStatus (HttpTransporter.java:544)
    at org.eclipse.aether.transport.http.HttpTransporter.execute (HttpTransporter.java:368)
    at org.eclipse.aether.transport.http.HttpTransporter.implPut (HttpTransporter.java:341)
    at org.eclipse.aether.spi.connector.transport.AbstractTransporter.put (AbstractTransporter.java:123)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:616)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:597)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:563)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:383)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:305)
    at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:301)
    at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:219)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:437)
    at org.apache.maven.plugins.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:156)
    at org.apache.maven.plugins.deploy.DeployMojo.execute (DeployMojo.java:194)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke (Method.java:578)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/8.4/foobar-8.4.pom (1.3 kB at 6.1 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/8.4/foobar-8.4.jar
[WARNING] Failed to upload checksum to gl/pru/foobar/8.4/foobar-8.4.jar.sha1
org.apache.http.client.HttpResponseException: status code: 404, reason phrase: Not Found (404)
    at org.eclipse.aether.transport.http.HttpTransporter.handleStatus (HttpTransporter.java:544)
    at org.eclipse.aether.transport.http.HttpTransporter.execute (HttpTransporter.java:368)
    at org.eclipse.aether.transport.http.HttpTransporter.implPut (HttpTransporter.java:341)
    at org.eclipse.aether.spi.connector.transport.AbstractTransporter.put (AbstractTransporter.java:123)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:616)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:597)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:563)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:383)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:305)
    at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:301)
    at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:219)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:437)
    at org.apache.maven.plugins.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:156)
    at org.apache.maven.plugins.deploy.DeployMojo.execute (DeployMojo.java:194)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke (Method.java:578)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/8.4/foobar-8.4.jar (2.1 kB at 13 kB/s)
Downloading from gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml
Downloaded from gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml (554 B at 11 kB/s)
Uploading to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml
[WARNING] Failed to upload checksum to gl/pru/foobar/maven-metadata.xml.sha1
org.apache.http.client.HttpResponseException: status code: 409, reason phrase: Conflict (409)
    at org.eclipse.aether.transport.http.HttpTransporter.handleStatus (HttpTransporter.java:544)
    at org.eclipse.aether.transport.http.HttpTransporter.execute (HttpTransporter.java:368)
    at org.eclipse.aether.transport.http.HttpTransporter.implPut (HttpTransporter.java:341)
    at org.eclipse.aether.spi.connector.transport.AbstractTransporter.put (AbstractTransporter.java:123)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksum (BasicRepositoryConnector.java:616)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.uploadChecksums (BasicRepositoryConnector.java:597)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$PutTaskRunner.runTask (BasicRepositoryConnector.java:563)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:383)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.put (BasicRepositoryConnector.java:320)
    at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:332)
    at org.eclipse.aether.internal.impl.DefaultDeployer.deploy (DefaultDeployer.java:219)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.deploy (DefaultRepositorySystem.java:437)
    at org.apache.maven.plugins.deploy.AbstractDeployMojo.deploy (AbstractDeployMojo.java:156)
    at org.apache.maven.plugins.deploy.DeployMojo.execute (DeployMojo.java:194)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke (Method.java:578)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Uploaded to gl_pru: http://gdk.test:8000/api/v4/projects/22/packages/maven/gl/pru/foobar/maven-metadata.xml (583 B at 3.5 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.199 s
[INFO] Finished at: 2023-03-17T15:22:58+01:00
[INFO] ------------------------------------------------------------------------
Upload done.
Removing credentials for root
All done.

💥 With the feature flag enabled

In a rails console:

Feature.enable(:packages_registry_maven_uploads_force_primary)

All the uploads are accepted and the errors go away ✅

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #362668 (closed)

Edited by David Fernandez

Merge request reports