GitLab issueshttps://gitlab.com/gitlab-org/gitlab/-/issues2024-03-04T21:01:13Zhttps://gitlab.com/gitlab-org/gitlab/-/issues/443358Maven metadata.xml corruption2024-03-04T21:01:13ZCleveland Bledsoe JrMaven metadata.xml corruption### Summary
A customer encountered an issue with a corrupt `maven-metadata.xml` file during a build. It was unclear on why or how the file became corrupted, so it would be challenging to attempt to reproduce. The intention of the issue ...### Summary
A customer encountered an issue with a corrupt `maven-metadata.xml` file during a build. It was unclear on why or how the file became corrupted, so it would be challenging to attempt to reproduce. The intention of the issue is to document it to determine if others are experiencing similar issues and determine how it may have ended up in this state.
Error in the build for a project:
```
Could not read metadata /.m2/repository/path/to/file/maven-metadata-xx.xml: no more data available - expected end tags </lastUpdated></versioning></metadata> to close start tag <lastUpdated> from line 79 and start tag <versioning> from line 5 and start tag <metadata> from line 2, parser stopped on TEXT seen ...<lastUpdated>20240213044202</la... @79:36
```
Note the missing closing tag below:
```
104 <version>1.1.1-RELEASE</version>
105 <version>2.2.1-RELEASE</version>
106 <version>3.3.1-RELEASE</version>
107 <version>4.4.1-RELEASE</version>
108 <version>5.5.1-RELEASE</version>
109 </versions>
110 <lastUpdated>20240213044202</la
```
### Steps to reproduce
No reproduction
### Example Project
See [ZD Ticket](https://gitlab.zendesk.com/agent/tickets/503680) (Internal) for reference
### What is the current *bug* behavior?
Corrupt Maven metadata files can prevent builds and proper usage of maven commands
### What is the expected *correct* behavior?
Maven metadata files should not become corrupt or have a mechanism to regenerate under certain conditions.
### Relevant logs and/or screenshots
<!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code
as it's tough to read otherwise. -->
### Output of checks
This bug happens on GitLab.com 16.10.0-pre 64f5301ea4f
### Workarounds
Delete a relevant package from the package registry. This seemingly re-creates the metadata (sync_worker?)
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/427151Gradle: can't use the URL-encoded project path2023-10-06T05:54:10ZDavid FernandezGradle: can't use the URL-encoded project path## :fire: Problem
When using `$ gradle`, we can't use the URL-encoded project path as the id of the project.
`$ gradle` will interpret those `%2F` into `/` and thus, it will not properly work.
`build.gradle` example:
```groovy
plugin...## :fire: Problem
When using `$ gradle`, we can't use the URL-encoded project path as the id of the project.
`$ gradle` will interpret those `%2F` into `/` and thus, it will not properly work.
`build.gradle` example:
```groovy
plugins {
id 'java'
id 'application'
}
repositories {
maven {
url "https://gitlab.com/api/v4/projects/issue-reproduce%2Fpackages%2Fmaven%2F332028-test/packages/maven"
name "GitLab"
credentials(HttpHeaderCredentials) {
name = 'Private-Token'
value = '<token>'
}
authentication {
header(HttpHeaderAuthentication)
}
}
}
dependencies {
implementation 'foo.bar.app:something-works:3.1'
}
```
## :crystal_ball: The available workaround
The available workaround here is to use the project `id` directly instead of its full path.
## :fire_engine: Solutions
There are different ways to solve this:
* Properly document that URL-encoded project full paths are not supported with `$ gradle`.
* Check if we could do something about it in the Grape API router.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/424238Database race condition when uploading maven packages2024-03-14T14:00:55ZCaleb WilliamsonDatabase race condition when uploading maven packages<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "type::bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://g...<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "type::bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
When using official maven images tagged with `eclipse-temurin`(e.g. `maven:3.9.3-eclipse-temurin-11`) it is possible to encounter a race condition when uploading packages, ultimately resulting in the following output within the job and a `{"message":"Validation failed: Name has already been taken"}` error within our logs:
```
Could not transfer artifact calebw:maven-test2:pom:0.59.40 from/to gitlab-maven (https://gitlab.com/api/v4/projects/12345678/packages/maven): status code: 400, reason phrase: Bad Request (400)
```
This seems to **only occur** when **both** of the following conditions are true:
- An official maven image tagged with `eclipse-temurin` is used
- The flag `-DdeployAtEnd=true` is passed as a Maven argument, which configures all the packages to be pushed at the end.
After speaking with @10io, it seems like this is occurring due to how these specific images send the requests when the above conditions are met. It looks like we receive two authorize requests sequentially to the `/authorize` endpoint (one for the pom, one for the jar) before a request to upload either one of these files is made.
### Steps to reproduce
Fork [this project](https://gitlab.com/calebw/mvn-400-bad-request) and run a pipeline. This project utilizes predefined CI variables and should need no configuration in order to build and attempt to publish packages to it's relevant package registry.
This project has a parent/child pom configuration that publishes multiple packages. You will see the job fail with the above error on a random package, and it will fail when attempting to publish either the POM or JAR file. Removing the `-DdeployAtEnd=true` flag from the `.gitlab-ci.ym` or changing the image to `maven:3.9.3` will allow the job to succeed.
Relevant logs can be found in Kibana after a failed attempt is made by searching on the following: `json.path: /api/v4/projects/PROJECT_ID/packages/maven/NAMESPACE/maven-test/*`.
### Example Project
https://gitlab.com/calebw/mvn-400-bad-request
### What is the current *bug* behavior?
Deploy attempts will fail with a `400 bad request` output in the job, and a `{"message":"Validation failed: Name has already been taken"}` error on the backend.
### What is the expected *correct* behavior?
All packages are deployed without issue.
### Relevant logs and/or screenshots
<!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code
as it's tough to read otherwise. -->
### Output of checks
<!-- If you are reporting a bug on GitLab.com, uncomment below -->
This bug happens on GitLab.com
### Possible fixes
Implement support for parallel uploads. This could be challenging but we could have some leads:
Use upsert this way, the database itself will take care of the race condition. The challenge here is that we need to have the proper unique constraints in place.
Use an exclusive lease on the backend so that second threads wait for the first one to finish. Not ideal but could work.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/409676Document basic auth for mvn and gradle2023-05-04T11:07:41ZDavid FernandezDocument basic auth for mvn and gradle## :fire: Problem
In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118692#note_1370366017, we added basic auth support to the download endpoints of the ~"Maven Repository".
This was to support downloads for the `$ sbt` client.
...## :fire: Problem
In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/118692#note_1370366017, we added basic auth support to the download endpoints of the ~"Maven Repository".
This was to support downloads for the `$ sbt` client.
In that MR, we updated the ~documentation for `$ sbt` but not `$ mvn` or `$ gradle`.
## :fire_engine: Solution
Update the ~"Maven Repository" ~documentation so that it is clearly noted that basic auth is support for pulling packages (not pushing).Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/404733Parallel Maven uploads lead to 409 Conflict responses2023-10-24T01:20:46ZDavid FernandezParallel Maven uploads lead to 409 Conflict responses## :fire: Problem
In https://gitlab.com/gitlab-org/gitlab/-/issues/367356#note_1341809603, we discovered that the ~"Maven Repository" could receive parallel publications for the exact same package name and version.
That could lead to u...## :fire: Problem
In https://gitlab.com/gitlab-org/gitlab/-/issues/367356#note_1341809603, we discovered that the ~"Maven Repository" could receive parallel publications for the exact same package name and version.
That could lead to upload the same file (eg. same filename) at the same time. The problem is that maven clients will upload a `.sha1` digest after upload a file.
We have these events:
1. Client A: uploads file `foobar.txt` for package `test`, version `1.2.3`.
1. Client B: uploads file `foobar.txt` (different contents) for package `test`, version `1.2.3`. This is now the most recent `foobar.txt` file. This is the file that will considered for checking `.sha1` digests.
1. Client B: uploads file `foobar.txt.sha1`. It matches the `sha1` of (2.), the backend will reply `200 Ok`
1. Client A: uploads file `foobar.txt.sha1`. This is the `sha1` of (1.). It doesn't match the `sha1` of (2.), the backend will return `409 Conflict`. :boom:
This issue is to discuss if the backend should handle (4.) in a different way or not.
### :stadium: Additional considerations
* Parallel maven publications is not a common thing to do but I can see it happening with 2 CI pipelines running and publishing the exact same package name+version.
* Maven already has a workaround for this situation: snapshot versions. Snapshot versions will add a suffix that is set by using a timestamp. This means that two parallel snapshot uploads can perfectly be handled.
* Is it technically wrong to answer `409` in 4? Well, the ~backend always work with the most recent files, _always_. If it receives a `sha1` from an older file, then there is something fishy (parallel upload) about the upload. It could be ok to keep it as it is.
* Measuring impact. At the time of this writing, in the last 24 hours, ~"gitlab.com" received:
* `72 538` successful `sha1` uploads (`204 Created`)
* `48` conflict errors in `sha1` uploads (`409 Conflict`)
* `409 Conflict` represents about `0.07%` of all uploads.
* Please note that the upload of the `.sha1` file will fail _but_ the publication maven command will still go on and even be successful. In other words, a failing `sha1` upload does not interrupt the maven package publication.
* Given the challenges on the possible solutions (see below), I'm not sure that this issue is worth solving.
## :fire_engine: Solution
Things are not simple as maven clients will not send an identifier to link all uploads of the same package publication. In fact, they do this but _only_ for snapshot versions.
Here are some leads for the solution:
* Inspect older files to see of the uploaded `sha1` exists.
* Use something like a lease so that parallel uploads can't happen.
The concerns I have with the above solutions is that we could have edge cases. Verifying a `.sha1` is a useful feature. When the backend returns an error for a `.sha1`, it indicates a very precise situation: my most recent file doesn't match the signature you sent. This means that we could have an issue with the most recent file.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/396661Improve Snapshot dynamic parts extraction2023-03-15T07:49:08ZDavid FernandezImprove Snapshot dynamic parts extraction## :fire: Problem
For the fix of https://gitlab.com/gitlab-org/gitlab/-/issues/325749, we had to build a way to extract the dynamic part that ~"Maven Repository" clients put on file names.
Example: `my-package-11.0-20230315.073119-2.po...## :fire: Problem
For the fix of https://gitlab.com/gitlab-org/gitlab/-/issues/325749, we had to build a way to extract the dynamic part that ~"Maven Repository" clients put on file names.
Example: `my-package-11.0-20230315.073119-2.pom`, `20230315.073119` is the dynamic part. It changes each time that `$ mvn deploy` runs.
To detect that part, we used a [regex](https://gitlab.com/gitlab-org/gitlab/-/blob/177c7a54a54ae643b289e2590af99724331769c2/lib/gitlab/regex.rb#L12). That one will match the very last match.
The problem is that users can use _classifiers_ to generate additional artifacts. They can use any string for the _classifier_ part.
Example: `my-package-11.0-20230315.073119-2-fancy-classifier.jar`, `20230315.073119` is the dynamic part and `fancy-classifier` is the classifier.
The regex will not work if the classifier follows the same structure as the dynamic part. For example, `my-package-11.0-20230315.073119-2-55555555.444444-3.jar`, here the backend will wrongly extract `55555555.444444-3` as the classifier.
## :fire_engine: Solution
Improve the classifier extraction:
* Remove the package name and version from the file name and match the first group with the regex.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/396660Document Maven version warning2023-03-15T07:49:08ZDavid FernandezDocument Maven version warning## :fire: Problem
From https://gitlab.com/gitlab-org/gitlab/-/issues/325749#note_1308829837: Maven doesn't allow specific characters in the version string of maven packages.
However, the build itself will not fail. This means that arti...## :fire: Problem
From https://gitlab.com/gitlab-org/gitlab/-/issues/325749#note_1308829837: Maven doesn't allow specific characters in the version string of maven packages.
However, the build itself will not fail. This means that artifacts with malformed versions can be pushed.
The GitLab ~"Maven Repository" [relies](https://gitlab.com/gitlab-org/gitlab/-/blob/92312944bb8a827b5d00509a5d07d0fe8ee4955d/app/services/packages/maven/find_or_create_package_service.rb#L39) on a structure format to extract the package name and the package version. Malformed versions will end up in a buggy behavior for that extraction.
## :fire_engine: Solution
* Document that the GitLab ~"Maven Repository" doesn't support such malformed versionsBackloghttps://gitlab.com/gitlab-org/gitlab/-/issues/394813Maven package registry: move checks from upload to authorize endpoint2024-03-19T00:23:05ZDavid FernandezMaven package registry: move checks from upload to authorize endpoint## :fire: Problem
The ~"Maven Repository" uses direct uploads for its uploads. We're not going to detail all the aspects here but basically:
1. workhorse intercepts the upload.
2. workhorse asks rails: hey where do you want me to put th...## :fire: Problem
The ~"Maven Repository" uses direct uploads for its uploads. We're not going to detail all the aspects here but basically:
1. workhorse intercepts the upload.
2. workhorse asks rails: hey where do you want me to put this (request on the `/authorize` endpoint).
3. workhorse uploads the file to the location pointed by rails.
4. workhorse call the rails upload endpoint with a "file" parameter which is a "pointer" to the location where the file has been uploaded (request on the upload endpoint).
We see that rails receives `2` requests during the upload (step (2.) and (4.)).
Currently, step (2.) checks are fairly simple:
* check that the user has the right permissions.
* check that the workhorse header is present.
Then on step (4.), we have additional checks:
* `.md5` file + `FIPS`
* duplicates check (users can control if duplicates are allowed or not).
The issue is that those additional checks could be done in step (2.). By doing so, we would save step (3.) which is an interaction with object storage. Also that would mean one less thing to do for workhorse.
## :fire_engine: Solution
* Move the additional checks from steps (2.) and (4.)
* For step (4.), extract the duplication check into its own service.
* :warning: Use a ~"feature flag".Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/386260Maven package registry, switch S3 HEAD support from local to global2023-02-06T17:12:43ZDavid FernandezMaven package registry, switch S3 HEAD support from local to global### :fire: Problem
In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/27612, we patched a bug for the ~"Maven Repository": `HEAD` requests with an AWS S3 object storage didn't work (due to a signature mismatch).
At that time, we ...### :fire: Problem
In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/27612, we patched a bug for the ~"Maven Repository": `HEAD` requests with an AWS S3 object storage didn't work (due to a signature mismatch).
At that time, we patched the ~"Maven Repository" locally.
After 2 years+ of being implemented, we can consider the fix as stable and thus "promote" it into the file download helper function so that _all_ (for all APIs) file download endpoints are patched.
That's what https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106242 is doing.
### :fire_engine: Solution
For the ~"Maven Repository", we will need to switch from:
* a local patch _to_
* use the regular file download helper function (that contains the `HEAD` requests support)
Now, this only impacts ~"self-managed" users as ~"gitlab.com" runs on Google Cloud Storage (and not AWS S3).
Still this change can break the ~"Maven Repository" file download endpoints and that could break important user flows.
As such, we need to use a ~"feature flag" for this switch. Moreover, since ~"self-managed" users are only impacted, we will need to release this switch with the default enabled ~"feature flag".
Once, the change seems stable (for example, waiting a full milestone), remove the ~"feature flag".Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/378800Docs feedback: Maven archetype repo needs access_token parameter in URL2023-11-25T01:02:42ZKatrin Leinweber (GTLB)Docs feedback: Maven archetype repo needs access_token parameter in URLA ~"GitLab Ultimate" ~customer shared [in an internal ticket](https://gitlab.zendesk.com/agent/tickets/332347), that our ["… endpoint for Maven packages" doc](https://docs.gitlab.com/ee/user/packages/maven_repository/#use-the-gitlab-endp...A ~"GitLab Ultimate" ~customer shared [in an internal ticket](https://gitlab.zendesk.com/agent/tickets/332347), that our ["… endpoint for Maven packages" doc](https://docs.gitlab.com/ee/user/packages/maven_repository/#use-the-gitlab-endpoint-for-maven-packages) is missing details/example (or a whole section) specific to Maven archetype plugins and `archetype-catalog.xml` files.
The configuration we suggest leads to warning about `No archetype found in remote catalogue` in those cases.
The customer found it necessary to append `org/catalog/1.0/archetype-catalog.xml?access_token=<PRIVAET_TOKEN>&` to the `/packages/maven` URL:
```xml
<repositories>
<repository>
<id>archetype</id>
<url>https://gitlab.example.com/api/v4/projects/PROJECT_ID/packages/maven/org/catalog/1.0/archetype-catalog.xml?access_token=<PRIVAET_TOKEN>&</url>
```
to get it working. The `&` specifically was needed because:
> maven plugin auto add the "archetype-catalog.xml" as suffix, which we have to ignore.
<!-- Don't edit below this line -->BacklogMarcel AmiraultMarcel Amiraulthttps://gitlab.com/gitlab-org/gitlab/-/issues/375065Selected "Number of duplicate assets to keep" does not work with Maven Snapshots2023-12-02T01:42:22ZObjectway GmbHSelected "Number of duplicate assets to keep" does not work with Maven SnapshotsHi Gitlab Team,
I am unsure if the Implementation related to issue #352566 is working as intended. Reading the issue, I would have expected this feature to work with the Maven Snapshot files.
I have set the number of duplicated assets ...Hi Gitlab Team,
I am unsure if the Implementation related to issue #352566 is working as intended. Reading the issue, I would have expected this feature to work with the Maven Snapshot files.
I have set the number of duplicated assets to keep to one.
Regardless of the setting, our Maven Builds produce continuous Snapshot versions.
![image](/uploads/cd1b5c702a34aff4f7c769de15fa17a6/image.png)
Running the CI/CD Pipeline multiple times creates multiple assets. Here I would expect to keep only one and the latest SNAPSHOT assets.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/374571Cloning project over SSH fails when using gemnasium-python image2024-03-07T22:37:04ZDuncandrharris@gitlab.comCloning project over SSH fails when using gemnasium-python image### Summary
SSH checkout fails when using the Gemnasium-Python image
### Steps to reproduce
1. Configure a CI/CD job to check out a project via SSH using the `gemnasium-python:3` image
### Example Project
https://gitlab.com/gitlab-g...### Summary
SSH checkout fails when using the Gemnasium-Python image
### Steps to reproduce
1. Configure a CI/CD job to check out a project via SSH using the `gemnasium-python:3` image
### Example Project
https://gitlab.com/gitlab-gold/duncan/sshclone/
This project uses the same `before_script`, `script`, and project variables. The only change is job image.
### What is the current *bug* behavior?
SSH authentication fails with error `debug1: read_passphrase: can't open /dev/tty: No such device or address`
### What is the expected *correct* behavior?
Authentication should succeed.
### Relevant logs and/or screenshots
https://gitlab.com/gitlab-gold/duncan/sshclone/-/jobs/3042046163
```
$ ssh -Tvvv git@gitlab.com
OpenSSH_8.4p1 Debian-5+deb11u1, OpenSSL 1.1.1n 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2'
debug2: resolving "gitlab.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to gitlab.com [172.65.251.78] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-5+deb11u1
debug1: Remote protocol version 2.0, remote software version GitLab-SSHD
debug1: no match: GitLab-SSHD
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.com:22 as 'git'
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-s
debug2: host key algorithms: ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519,rsa-sha2-256,rsa-sha2-512,ssh-rsa
debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw
debug1: read_passphrase: can't open /dev/tty: No such device or address
Host key verification failed.
```
### Output of checks
This bug happens on GitLab.com
#### Results of GitLab environment info
GitLab Enterprise Edition 15.4.0-pre 8ef12c3857d
#### Results of GitLab application Check
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->BacklogYasha RiseYasha Risehttps://gitlab.com/gitlab-org/gitlab/-/issues/369059Maven Packages: The response body should be a String, but it is of type NilClass2024-03-19T00:23:17ZBotMaven Packages: The response body should be a String, but it is of type NilClasshttps://gitlab.com/gitlab-org/gitlab/-/blob/39d35cb97853f4b5dc5e7b46a9d911ec2f34b9d9/lib/api/api_guard.rb#L227 is catching this: the API endpoint is returning a `nil` body instead of a string.
https://sentry.gitlab.net/gitlab/gitlabcom/...https://gitlab.com/gitlab-org/gitlab/-/blob/39d35cb97853f4b5dc5e7b46a9d911ec2f34b9d9/lib/api/api_guard.rb#L227 is catching this: the API endpoint is returning a `nil` body instead of a string.
https://sentry.gitlab.net/gitlab/gitlabcom/issues/3352847/?referrer=gitlab_plugin
```
ArgumentError: The response body should be a String, but it is of type NilClass
```Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/362494Maven sync worker fails with invalid metadata.2023-09-12T18:02:24ZDavid FernandezMaven sync worker fails with invalid metadata.## :fire: Problem
We got an alarm on the Maven sync worker due to too many failures.
On [inspection](https://log.gprd.gitlab.net/goto/46908940-d4fb-11ec-aade-19e9974a7229) (internal), it seems that all failures are
```
metadata_conten...## :fire: Problem
We got an alarm on the Maven sync worker due to too many failures.
On [inspection](https://log.gprd.gitlab.net/goto/46908940-d4fb-11ec-aade-19e9974a7229) (internal), it seems that all failures are
```
metadata_content is invalid
Packages::Maven::Metadata::SyncWorker::SyncError
app/workers/packages/maven/metadata/sync_worker.rb:37:in `perform'
```
This indicates that the [`SyncService`](https://gitlab.com/gitlab-org/gitlab/-/blob/ca2318f4fa7c973c6b9ece259d763834d1d33355/app/services/packages/maven/metadata/sync_service.rb) is not successful in its execution.
We assume that the error comes from creating the versions xml file, given that the errors we see target android applications and we really doubt that this error comes from the plugin versions xml file.
So this [function](https://gitlab.com/gitlab-org/gitlab/-/blob/7baa6a66fe82c5ab49762a7a95ca2fc903922eb6/app/services/packages/maven/metadata/create_versions_xml_service.rb#L40-42) seems to fail = the expected xml nodes are not present in the metadata file.
[Kibana search](https://log.gprd.gitlab.net/goto/82b564a0-d518-11ec-aade-19e9974a7229).
## :fire_engine: Solution
* Get a metadata file content from an existing package that produces the error.
* Investigate how we can handle this situation instead of making the work fail.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/354820Discussion: should the maven-metadata.xml be shown on the package UI2022-10-17T00:27:14ZKatie MacoyDiscussion: should the maven-metadata.xml be shown on the package UI## Context
When the user allows Maven duplicates, each publish to the registry adds
* the `maven-metadata.xml` to the versionless package (package with *no* version)
* the .pom and .jar files to the package with a version.
When consumi...## Context
When the user allows Maven duplicates, each publish to the registry adds
* the `maven-metadata.xml` to the versionless package (package with *no* version)
* the .pom and .jar files to the package with a version.
When consuming the package, only the latest `maven-metadata.xml` is served. The user cannot access the older `maven-metadata.xml` files on the UI or API.
## A note about snapshots
Currently Maven snapshots do show the `maven-metadata.xml` in the UI.
Snapshot packages don't have `1` but `2` `maven-metadata.xml` files:
1. One in the versionless package (<- this one exists for *all* maven packages)
1. One in the package with the version (<- this one only exists for snapshot versions)
(1.) is not visible in the UI.
(2.) being in the package with the version, you can see it in the UI.
## Question
Should we show the (versionless package) `maven-metadata.xml` on the UI?
## User should not be able to delete the `maven-metadata.xml`
Note: if we show the file the UI, we should not allow the user to delete it.
This file is technically the *state* of the package registry for a given package. This is used during the requests :ping_pong: with `$ mvn` when `$ mvn` asks: `Hey third party registry, which versions of package foobar are available?`. This is when the ~backend returns the most recent `maven-metadata.xml` file.
Because of the above, if we allow the delete operation on that file, this can have a pretty deep impact on the registry behavior for that package (eg. `$ mvn` could complain that a specific version doesn't exist on the registry).
## Options
1. we show the file the UI but do not allow delete operation - consistent with competitors, maybe more clear what is being stored as part of the package.
1. we continue to not show it. - I have not seen any customer complaints about not showing it.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/354665Overwrite maven-metadata.xml file2023-04-10T17:32:54ZKatie MacoyOverwrite maven-metadata.xml file## Problem
When the user allows Maven duplicates, each publish to the registry adds
* the `maven-metadata.xml` to the versionless package (package with _no_ version)
* the .pom and .jar files to the package with a version.
_Versionles...## Problem
When the user allows Maven duplicates, each publish to the registry adds
* the `maven-metadata.xml` to the versionless package (package with _no_ version)
* the .pom and .jar files to the package with a version.
_Versionless_ packages are not returned by APIs so the .pom and .jar are displayed in the UI but the `maven-metadata.xml` is not.
When consuming the package, only the latest `maven-metadata.xml` is served. The user cannot access the older `maven-metadata.xml` files.
With time, a versionless maven package can accumulate _many_ `maven-metadata.xml` files. This consumes object storage space.
## Proposal
Overwriting the `maven-metadata.xml` with each publish of a duplicate Maven package to the registry would reduce storage space, since the older versions of the file are not accessible anyway. This is consistent with what Artifactory does.
## Other considerations
With cleanup policies for packages, the _versionless_ package will be taken into account.
Example, let's say that users want to keep only 10 of the duplicated package files. The _versionless_ package will be cleaned up as any other package.
Still, we know for sure that only the most recent file is used in the operations with `$ mvn` or `$ gradle`.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/343793Enhance API to correctly reflect state of the package registry in the maven-m...2023-06-20T13:16:16ZBruce MeierEnhance API to correctly reflect state of the package registry in the maven-metadata.xml file.**Background**:
We have been migrating maven artifacts from nexus to package registry.
- It uses the maven dependency:get to retrieve artifacts from nexus.
- It uses the gitlab REST api to publish artifacts to the package registry.
...**Background**:
We have been migrating maven artifacts from nexus to package registry.
- It uses the maven dependency:get to retrieve artifacts from nexus.
- It uses the gitlab REST api to publish artifacts to the package registry.
In nexus, we see a maven-metadata.xml (I have omitted the releases between 0.0.4 and 2.0.39 for brevity).
\<metadata\>
\<groupId\>redactedGroup\</groupId\>
\<artifactId\>redactedProject1\</artifactId\>
\<versioning\>
\<latest\>1.0.7\</latest\>
\<release\>2.0.42\</release\>
\<versions\>
\<version\>0.0.1\</version>
\<version\>0.0.3\</version>
\<version\>0.0.4\</version>
...
\<version\>2.0.39\</version>
\<version\>2.0.40\</version>
\<version\>2.0.41\</version>
\<version\>2.0.42\</version>
\</versions\>
\<lastUpdated\>20210909140329\</lastUpdated>
\</versioning\>
\</metadata\>
And of course, nexus contains these artifacts as well.
After migrating a number of artifacts from nexus to the package registry, we attempt to look at maven-metadata.xml via this url
https://redacted/packages/maven/redactedGroup/redactedProject1/maven-metadata.xml and we get a 404, file not found.
We can contrast that with a project where we have published artifacts to the gitlab package registry using maven’s release plugin. If we use a similar url:
https://redacted/packages/maven/redactedGroup/redactedProject2/maven-metadata.xml
\<metadata\>
\<groupId\>redactedGroup\</groupId\>
\<artifactId\>redactedProject2\</artifactId\>
\<versioning\>
\<release\>1.0.41\</release\>
\<versions\>
\<version\>1.0.24\</version\>
\<version\>1.0.25\</version\>
\<version\>1.0.26\</version\>
\<version\>1.0.28\</version\>
\<version\>1.0.29\</version\>
\<version\>1.0.30\</version\>
\<version\>1.0.36\</version\>
\<version\>1.0.37\</version\>
\<version\>1.0.38\</version\>
\<version\>1.0.39\</version\>
\<version\>1.0.40\</version\>
\<version\>1.0.41-SNAPSHOT\</version\>
\<version\>1.0.41\</version\>
\<version\>1.0.42-SNAPSHOT\</version\>
\</versions\>
\<lastUpdated\>20211005154204\</lastUpdated\>
\</versioning\>
\</metadata\>
**Desired Outcome:**
- We would like the artifacts which are published to the package registry via the web api to accurately be reflected in the maven-metadata.xml files.
- Or does gitlab have a url to rebuild maven-metadata.xml?
- It appears that the maven-metadata.xml is rebuilt when a package is removed. See https://gitlab.com/gitlab-org/gitlab/-/issues/11424. Could this functionality be triggered by an REST api call?Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/343530Intermittent issues with publishing maven packages to the registry2023-10-24T01:47:50ZKate GrechishkinaIntermittent issues with publishing maven packages to the registry<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab....<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
Sometimes, when publishing maven packages to GitLab maven package registry, the `PUT` event results in 500 error which fails the whole pipeline. This can be quite frustrating because a deploy job needs to be manually restarted to complete the deployment. When retrying the same pipeline with zero changes, the same package can be successfully deployed to the maven registry.
Searching through the logs, I have found ~200 events of 500 errors for the past 7 days:
```
Rack::Timeout::RequestTimeoutException
Request ran for longer than 60000ms
```
### Steps to reproduce
The issue is transient, no steps to reproduce, unfortunately, but customers that heavily use maven registry encounter it quite often. One of such customers has contacted us in [this ticket](https://gitlab.zendesk.com/agent/tickets/245891) to raise our awareness.
### Example Project
You can find the projects that experience this issue by running this query in Kibana:
https://log.gprd.gitlab.net/goto/90677fb8197acadf455753ed2de07e22
### What is the current *bug* behavior?
Intermittent 500 errors when publishing packages to maven repository
### What is the expected *correct* behavior?
Users don't receive intermittent 500 errors when publishing packages to maven repos.
### Relevant logs and/or screenshots
```
app/uploaders/object_storage.rb:404:in `block in cache_remote_file!',
app/uploaders/object_storage.rb:403:in `tap',
app/uploaders/object_storage.rb:403:in `cache_remote_file!',
app/uploaders/object_storage.rb:367:in `cache!',
app/services/packages/create_package_file_service.rb:12:in `execute',
lib/api/maven_packages.rb:276:in `block (2 levels) in <class:MavenPackages>',
ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call',
ee/lib/gitlab/ip_address_state.rb:10:in `with',
ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call',
lib/api/api_guard.rb:213:in `call',
ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase',
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call',
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call',
lib/gitlab/middleware/speedscope.rb:13:in `call',
lib/gitlab/request_profiler/middleware.rb:17:in `call',
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call',
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call',
lib/gitlab/metrics/web_transaction.rb:46:in `run',
lib/gitlab/metrics/rack_middleware.rb:16:in `call',
lib/gitlab/jira/middleware.rb:19:in `call',
lib/gitlab/middleware/go.rb:20:in `call',
lib/gitlab/etag_caching/middleware.rb:21:in `call',
lib/gitlab/middleware/multipart.rb:178:in `block in call',
lib/gitlab/middleware/multipart.rb:63:in `with_open_files',
lib/gitlab/middleware/multipart.rb:177:in `call',
lib/gitlab/middleware/read_only/controller.rb:50:in `call',
lib/gitlab/middleware/read_only.rb:18:in `call',
lib/gitlab/middleware/same_site_cookies.rb:27:in `call',
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call',
lib/gitlab/middleware/basic_health_check.rb:25:in `call',
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call',
lib/gitlab/middleware/request_context.rb:21:in `call',
config/initializers/fix_local_cache_middleware.rb:11:in `call',
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call',
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call',
lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call',
lib/gitlab/middleware/release_env.rb:12:in `call'
```
### Output of checks
This bug happens on GitLab.com
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/339112`Packages::Maven::FindOrCreatePackageService` don't use a transaction2021-11-30T18:40:17ZDavid Fernandez`Packages::Maven::FindOrCreatePackageService` don't use a transaction### :fire: Problem
https://gitlab.com/gitlab-org/gitlab/-/blob/3177ec13ff15b9111aef4e7d6ee2097ef9612e0a/app/services/packages/maven/find_or_create_package_service.rb is not using a single transaction to contain all the write operations....### :fire: Problem
https://gitlab.com/gitlab-org/gitlab/-/blob/3177ec13ff15b9111aef4e7d6ee2097ef9612e0a/app/services/packages/maven/find_or_create_package_service.rb is not using a single transaction to contain all the write operations.
As such, many transactions (one per write) is created. The problem is that if one of the write operation fails, the others are not rolled back = we have non coherent data.
### :fire_engine: Solution
The `#execute` function should open a transaction for all the underlying logic. Then each write will automatically join this transaction if needed.
```ruby
def execute
project.transaction do
# rest of the actual code here.
end
end
```
:warning: DO NOT USE A NEW SUB TRANSACTION (`project.transaction(requires_new: true)`). This is known to have ~performance issues, see https://gitlab.com/gitlab-org/gitlab/-/issues/338346.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/330936Display the package metadata for Maven, RubyGems, and Composer2021-08-25T13:52:02ZTim Rizzitrizzi@gitlab.comDisplay the package metadata for Maven, RubyGems, and Composer### Context
In the epic https://gitlab.com/groups/gitlab-org/-/epics/6007 the Package group has been adding the contents of a packages metadata file to the user interface on the package details page. This issue focuses on adding data fo...### Context
In the epic https://gitlab.com/groups/gitlab-org/-/epics/6007 the Package group has been adding the contents of a packages metadata file to the user interface on the package details page. This issue focuses on adding data for Maven, RubyGems and Composer packages.
### Proposal
When Maven, RubyGems and Composer packages are uploaded to GitLab the files containing their respective metadata is directly stored in object storage as a `PackageFile` or is stored directly in the DB. This data will be made available to the front end to help users find and validate their dependencies.
#### Further details
##### Front end considerations
* The file explorer in gitlab uses the `blob-viewer` which expects a `blob` passed in input
* The blob is returned by this graphql type => `app/graphql/types/repository/blob_type.rb`
If we could have the ~backend return a BlobType for our files we could on the ~frontend re use the `blob-viewer` with the following perks
* Consistent UI
* Syntax highlighting
* Do not add more context
* Easily switch to `edit mode` in the futureBacklog