Maven: 400 Bad Request uploading packages to GitLab - issue introduced in 12.10.14

Summary

Customers are reporting that when they upgrade from 12.10.x to 12.10.14, Maven package uploads to the GitLab package repo fail with 400 Bad Request

  • Customers are confirming that downgrading to 12.10.13 resolves the issue for them.

This was also encountered internally doing testing for another ticket.

  • Downgrading GitLab to 12.10.13 fixed the issue - ie, re-running the exact same job succeeds.

Workaround

  • Downgrade to GitLab 12.10.13
  • Upgrade to GitLab 13.0.9 or higher as soon as possible.

Work required

As 12.10 is now not receiving bug fix updates, this issue is tracking and documenting the regression in 12.10.14

When the issue becomes scheduled for work, please investigate why the regression was not identified prior to 12.10.14 being released. Automatic testing should have identified that pushing Maven packages was not possible?

Customer tickets (internal)

Steps to reproduce

  • Deploy example project to 12.10.13
  • Run the manual maven repository job.
  • It should succeed, and upload a small package to GitLab.
  • Upgrade to 12.10.14
  • Re-run the job. It'll fail.

Example Project

https://gitlab.com/bprescott-support/testing/issue229482-badrequest400

(This doesn't reproduce on GitLab.com, but this was pushed from a project that reproduces it on 12.10.14)

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

See attachments linked above for full output

12.10.13

[INFO] 
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ maven-smoke-test ---
Downloading from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-metadata.xml
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.jar
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.jar (2.8 kB at 1.9 kB/s)
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.pom
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.pom (3.2 kB at 3.7 kB/s)
Downloading from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml
Downloaded from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml (356 B at 890 B/s)
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-metadata.xml
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-metadata.xml (785 B at 1.1 kB/s)
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml (399 B at 570 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  19.218 s
[INFO] Finished at: 2020-07-16T07:34:51Z
[INFO] ------------------------------------------------------------------------
Job succeeded

12.10.14

 [INFO] 
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ maven-smoke-test ---
Downloading from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-metadata.xml
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-smoke-test-1.0.3915-20200716.072311-1.jar
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-smoke-test-1.0.3915-20200716.072311-1.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16.389 s
[INFO] Finished at: 2020-07-16T07:23:12Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on project maven-smoke-test: Failed to deploy artifacts: Could not transfer artifact com.gitlab:maven-smoke-test:jar:1.0.3915-20200716.072311-1 from/to gitlab-maven (https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven): Transfer failed for https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-smoke-test-1.0.3915-20200716.072311-1.jar 400 Bad Request -> [Help 1]
[ERROR]

Output of checks

Results of GitLab environment info

This is run on 12.10.13; the only change made was yum downgrade gitlab-ee-12.10.13

Expand for output related to GitLab environment info 


System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.7
Git Version:	2.26.2
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.10.13-ee
Revision:	ec145a32727
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://bprescott-gitlabtest-0.do.gitlap.com
HTTP Clone URL:	https://bprescott-gitlabtest-0.do.gitlap.com/some-group/some-project.git
SSH Clone URL:	git@bprescott-gitlabtest-0.do.gitlap.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	12.2.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
2/3 ... yes
2/4 ... yes
2/6 ... yes
2/7 ... yes
4/8 ... yes
2/9 ... yes
2/10 ... yes
2/11 ... yes
2/12 ... yes
2/13 ... yes
2/14 ... yes
6/15 ... yes
6/16 ... yes
2/17 ... yes
8/19 ... yes
8/20 ... yes
8/21 ... yes
4/22 ... yes
4/23 ... yes
2/24 ... yes
14/26 ... yes
14/27 ... yes
2/28 ... yes
2/29 ... yes
15/31 ... yes
15/32 ... yes
15/33 ... yes
2/34 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 3
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Support identified the follow as the root cause:

Edited by Tim Rizzi