Skip to content

Maven request forwarding

Steve Abrams requested to merge maven-forwarding into master

🐦 What does this MR do and why?

Users have the ability to publish and install Maven packages to the package registry. Maven developers generally use a configuration file local to where they are working that has a variety of remotes set for fetching various packages, similar to having different Git remotes set for different Git repositories.

The public registry for maven packages is maven central. This is where users will pull standard public packages from.

Having to set and maintain different remotes can become a hassle, so here we introduce Maven request forwarding. This allows users to set GitLab as their single Maven repository remote. They can then install any private packages on GitLab, and if they request a package that does not exist on GitLab, we reply with a redirect to the maven-central repository.

This is also the first step towards a broader set of features where we stop redirecting and start proxying the request within GitLab, eventually allowing us to cache these public dependencies for greater control and security. Those are a ways in the future, but should give some context on where this feature is going.

Note: this MR is implemented behind a feature flag

📸 Screenshots or screen recordings

Feature flag disabled 
~/workspace/playground/maven/maven-forwarding-project2 mvn install -U -s settings.xml
[INFO] Scanning for projects...
[INFO]
[INFO] -------------< com.gitlab.test:maven-forwarding-project2 >--------------
[INFO] Building maven-forwarding-project2 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.11/junit-4.11.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.402 s
[INFO] Finished at: 2022-09-08T23:39:52-06:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project maven-forwarding-project2: Could not resolve dependencies for project com.gitlab.test:maven-forwarding-project2:jar:1.0-SNAPSHOT: Failed to collect dependencies at junit:junit:jar:4.11: Failed to read artifact descriptor for junit:junit:jar:4.11: Could not transfer artifact junit:junit:pom:4.11 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [gitlab-maven (http://gdk.test:3000/api/v4/projects/32/packages/maven, default, releases+snapshots)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
Feature flag enabled for group using project-level endpoint 
~/workspace/playground/maven/maven-forwarding-project2 rm -rf ~/.m2/repository/junit
~/workspace/playground/maven/maven-forwarding-project2 rm -rf target
~/workspace/playground/maven/maven-forwarding-project2 mvn install -U -s settings.xml
[INFO] Scanning for projects...
[INFO]
[INFO] -------------< com.gitlab.test:maven-forwarding-project2 >--------------
[INFO] Building maven-forwarding-project2 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.11/junit-4.11.pom
Downloaded from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.11/junit-4.11.pom (2.3 kB at 3.7 kB/s)
Downloading from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.11/junit-4.11.jar
Downloaded from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.11/junit-4.11.jar (245 kB at 510 kB/s)
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ maven-forwarding-project2 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ maven-forwarding-project2 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ maven-forwarding-project2 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ maven-forwarding-project2 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ maven-forwarding-project2 ---
Downloading from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.12/junit-4.12.pom
Downloaded from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.12/junit-4.12.pom (24 kB at 42 kB/s)
Downloading from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.12/junit-4.12.jar
Downloaded from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/4.12/junit-4.12.jar (315 kB at 863 kB/s)
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.gitlab.test.AppTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in com.gitlab.test.AppTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ maven-forwarding-project2 ---
[INFO] Building jar: /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/maven-forwarding-project2-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ maven-forwarding-project2 ---
Downloading from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/3.8.1/junit-3.8.1.pom
Downloaded from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/3.8.1/junit-3.8.1.pom (998 B at 3.1 kB/s)
Downloading from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/3.8.1/junit-3.8.1.jar
Downloaded from central-proxy: http://gdk.test:3000/api/v4/projects/32/packages/maven/junit/junit/3.8.1/junit-3.8.1.jar (121 kB at 340 kB/s)
[INFO] Installing /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/maven-forwarding-project2-1.0-SNAPSHOT.jar to /Users/steveabrams/.m2/repository/com/gitlab/test/maven-forwarding-project2/1.0-SNAPSHOT/maven-forwarding-project2-1.0-SNAPSHOT.jar
[INFO] Installing /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/pom.xml to /Users/steveabrams/.m2/repository/com/gitlab/test/maven-forwarding-project2/1.0-SNAPSHOT/maven-forwarding-project2-1.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.983 s
[INFO] Finished at: 2022-09-08T23:39:05-06:00
[INFO] ------------------------------------------------------------------------
Feature flag enabled for group using group-level endpoint 
~/workspace/playground/maven/maven-forwarding-project2 rm -rf ~/.m2/repository/junit
~/workspace/playground/maven/maven-forwarding-project2 rm -rf target
~/workspace/playground/maven/maven-forwarding-project2 mvn install -U -s settings.xml
[INFO] Scanning for projects...
[INFO]
[INFO] -------------< com.gitlab.test:maven-forwarding-project2 >--------------
[INFO] Building maven-forwarding-project2 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.11/junit-4.11.pom
Downloaded from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.11/junit-4.11.pom (2.3 kB at 329 B/s)
Downloading from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.11/junit-4.11.jar
Downloaded from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.11/junit-4.11.jar (245 kB at 399 kB/s)
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ maven-forwarding-project2 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ maven-forwarding-project2 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ maven-forwarding-project2 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ maven-forwarding-project2 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ maven-forwarding-project2 ---
Downloading from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.12/junit-4.12.pom
Downloaded from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.12/junit-4.12.pom (24 kB at 77 kB/s)
Downloading from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.12/junit-4.12.jar
Downloaded from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/4.12/junit-4.12.jar (315 kB at 783 kB/s)
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.gitlab.test.AppTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in com.gitlab.test.AppTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ maven-forwarding-project2 ---
[INFO] Building jar: /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/maven-forwarding-project2-1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ maven-forwarding-project2 ---
Downloading from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/3.8.1/junit-3.8.1.pom
Downloaded from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/3.8.1/junit-3.8.1.pom (998 B at 3.0 kB/s)
Downloading from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/3.8.1/junit-3.8.1.jar
Downloaded from central-proxy: http://gdk.test:3000/api/v4/groups/92/-/packages/maven/junit/junit/3.8.1/junit-3.8.1.jar (121 kB at 277 kB/s)
[INFO] Installing /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/target/maven-forwarding-project2-1.0-SNAPSHOT.jar to /Users/steveabrams/.m2/repository/com/gitlab/test/maven-forwarding-project2/1.0-SNAPSHOT/maven-forwarding-project2-1.0-SNAPSHOT.jar
[INFO] Installing /Users/steveabrams/workspace/playground/maven/maven-forwarding-project2/pom.xml to /Users/steveabrams/.m2/repository/com/gitlab/test/maven-forwarding-project2/1.0-SNAPSHOT/maven-forwarding-project2-1.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10.493 s
[INFO] Finished at: 2022-09-08T23:38:40-06:00
[INFO] ------------------------------------------------------------------------
New setting available in the admin UI
Feature flag enabled Feature flag disabled
Screen_Shot_2022-09-13_at_11.29.07_AM Screen_Shot_2022-09-13_at_11.32.38_AM

💻 How to set up and validate locally

  1. Create a local maven project:

    mvn archetype:generate -DgroupId=com.gitlab.test -DartifactId=maven-forwarding-project -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false -DarchetypeVersion=1.4

  2. Navigate to the new project:

    cd maven-forwarding-project

  3. Create a settings.xml with your personal access token to authenticate with GitLab and include a mirrors section to point maven-central requests to GitLab and ensure the project never makes a request to maven-central:

    <settings>
  <servers>
    <server>
      <id>central-proxy</id>
      <configuration>
        <httpHeaders>
          <property>
            <name>Private-Token</name>
            <value>{token}</value>
          </property>
        </httpHeaders>
      </configuration>
    </server>
  </servers>
  <mirrors>
    <mirror>
      <id>central-proxy</id>
      <name>GitLab proxy of central repo</name>
      <url>http://gdk.test:3000/api/v4/projects/{project_id}packages/maven</url>
      <mirrorOf>central</mirrorOf>
    </mirror>
  </mirrors>
</settings>

  4. Install dependencies (the project has a bunch of default dependencies so you don't need to add any):

    mvn install -U -s settings.xml

  5. The terminal output should show requests going to GitLab, your requests may be include different dependencies in different orders:

    mvn install -U -s settings.xml
[INFO] Scanning for projects...
[INFO]
[INFO] --------------< com.gitlab.test:maven-forwarding-project >--------------
[INFO] Building maven-forwarding-project 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from central-proxy: http://gdk.test:3001/api/v4/projects/31/packages/maven/junit/junit/4.11/junit-4.11.pom
Downloaded from central-proxy: http://gdk.test:3001/api/v4/projects/31/packages/maven/junit/junit/4.11/junit-4.11.pom (0 B at 0 B/s)
Downloading from central-proxy: http://gdk.test:3001/api/v4/projects/31/packages/maven/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom
Downloaded from central-proxy: http://gdk.test:3001/api/v4/projects/31/packages/maven/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom (0 B at 0 B/s)
Downloading from central-proxy: http://gdk.test:3001/api/v4/projects/31/packages/maven/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom
Downloaded from central-proxy: http://gdk.test:3001/api/v4/projects/31/packages/maven/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom (0 B at 0 B/s)

...

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  30.111 s
[INFO] Finished at: 2022-04-16T20:55:39-06:00
[INFO] ------------------------------------------------------------------------

  6. To repeat the install you need to clear your local repository. I suggest just removing the junit dependency and the project target/ directory:

    rm -rf target
rm -rf ~/.m2/repository/junit

  7. Now you can retry the install but with the other endpoints or other Feature Flag configurations. Less dependencies will be fetched than the first time because we only reset the junit dependency.

🛃 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related: #362657 (closed)

Edited by Steve Abrams

Merge request reports