Maven: 400 Bad Request uploading packages to GitLab - issue introduced in 12.10.14
Summary
Customers are reporting that when they upgrade from 12.10.x to 12.10.14, Maven package uploads to the GitLab package repo fail with 400 Bad Request
- Customers are confirming that downgrading to 12.10.13 resolves the issue for them.
This was also encountered internally doing testing for another ticket.
- Downgrading GitLab to 12.10.13 fixed the issue - ie, re-running the exact same job succeeds.
Workaround
- Downgrade to GitLab 12.10.13
- Upgrade to GitLab 13.0.9 or higher as soon as possible.
Work required
As 12.10 is now not receiving bug fix updates, this issue is tracking and documenting the regression in 12.10.14
When the issue becomes scheduled for work, please investigate why the regression was not identified prior to 12.10.14 being released. Automatic testing should have identified that pushing Maven packages was not possible?
Customer tickets (internal)
- https://gitlab.zendesk.com/agent/tickets/164237
- https://gitlab.zendesk.com/agent/tickets/164687
- https://gitlab.zendesk.com/agent/tickets/165867
- https://gitlab.zendesk.com/agent/tickets/163678
Steps to reproduce
- Deploy example project to 12.10.13
- Run the manual maven repository job.
- It should succeed, and upload a small package to GitLab.
- Upgrade to 12.10.14
- Re-run the job. It'll fail.
Example Project
https://gitlab.com/bprescott-support/testing/issue229482-badrequest400
(This doesn't reproduce on GitLab.com, but this was pushed from a project that reproduces it on 12.10.14)
What is the current bug behavior?
What is the expected correct behavior?
Relevant logs and/or screenshots
See attachments linked above for full output
12.10.13
[INFO]
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ maven-smoke-test ---
Downloading from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-metadata.xml
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.jar
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.jar (2.8 kB at 1.9 kB/s)
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.pom
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-smoke-test-1.0.3921-20200716.073447-1.pom (3.2 kB at 3.7 kB/s)
Downloading from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml
Downloaded from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml (356 B at 890 B/s)
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-metadata.xml
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3921-SNAPSHOT/maven-metadata.xml (785 B at 1.1 kB/s)
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml
Uploaded to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/maven-metadata.xml (399 B at 570 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19.218 s
[INFO] Finished at: 2020-07-16T07:34:51Z
[INFO] ------------------------------------------------------------------------
Job succeeded
12.10.14
[INFO]
[INFO] --- maven-deploy-plugin:2.8.2:deploy (default-deploy) @ maven-smoke-test ---
Downloading from gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-metadata.xml
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-smoke-test-1.0.3915-20200716.072311-1.jar
Uploading to gitlab-maven: https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-smoke-test-1.0.3915-20200716.072311-1.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16.389 s
[INFO] Finished at: 2020-07-16T07:23:12Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on project maven-smoke-test: Failed to deploy artifacts: Could not transfer artifact com.gitlab:maven-smoke-test:jar:1.0.3915-20200716.072311-1 from/to gitlab-maven (https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven): Transfer failed for https://bprescott-gitlabtest-0.do.gitlap.com/api/v4/projects/22/packages/maven/com/gitlab/maven-smoke-test/1.0.3915-SNAPSHOT/maven-smoke-test-1.0.3915-20200716.072311-1.jar 400 Bad Request -> [Help 1]
[ERROR]
Output of checks
Results of GitLab environment info
This is run on 12.10.13; the only change made was yum downgrade gitlab-ee-12.10.13
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.26.2 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.10.13-ee Revision: ec145a32727 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: https://bprescott-gitlabtest-0.do.gitlap.com HTTP Clone URL: https://bprescott-gitlabtest-0.do.gitlap.com/some-group/some-project.git SSH Clone URL: git@bprescott-gitlabtest-0.do.gitlap.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 12.2.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes 2/3 ... yes 2/4 ... yes 2/6 ... yes 2/7 ... yes 4/8 ... yes 2/9 ... yes 2/10 ... yes 2/11 ... yes 2/12 ... yes 2/13 ... yes 2/14 ... yes 6/15 ... yes 6/16 ... yes 2/17 ... yes 8/19 ... yes 8/20 ... yes 8/21 ... yes 4/22 ... yes 4/23 ... yes 2/24 ... yes 14/26 ... yes 14/27 ... yes 2/28 ... yes 2/29 ... yes 15/31 ... yes 15/32 ... yes 15/33 ... yes 2/34 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.26.2) Git user has default SSH configuration? ... yes Active users: ... 3 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Support identified the follow as the root cause:
- changelog -lo1;§x