Content of all release source code asset tarballs has changed
I've just noticed a lot of build failures related to a changed checksums of all release tarballs for the projects hosted on gitlab.com, for example:
https://gitlab.com/prpl-foundation/components/ambiorix/applications/amx-cli/-/archive/v0.2.22/amx-cli-v0.2.22.tar.gz had for past 10 months 3faa2d50190524baa59f73d2af322d8c6a27be882d8d2bddd2e80384d8b93eb0
sha256sum hash, but the hash has changed recently to 6f8b753736e6badbcecd11952c6e6691934ebd4fda1f598dfad91855dbf6148f
.
It seems, that the hash of the release archives has changed due to the rename of the root directory in the tar archive, it has changed from previous amx-cli-v0.2.22
to current amx-cli-18c40c6e7e8b5998da52bcf56dbfe549594b76a7
.
Maybe related https://github.blog/changelog/2023-01-30-git-archive-checksums-may-change/ ?
Designs
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- Petr Štetiar mentioned in issue gitaly#4982 (closed)
mentioned in issue gitaly#4982 (closed)
- Developer
@ynezz Thanks for creating this issue! I've gotten similar reports yesterday, so let me have a look what has changed recently.
- Petr Štetiar changed the description
Compare with previous version changed the description
- Patrick Steinhardt assigned to @pks-gitlab
assigned to @pks-gitlab
- Patrick Steinhardt added priority2 label
added priority2 label
Did Gitlab update git and create another https://github.com/orgs/community/discussions/46034 ?
Collapse replies - Developer
@Neumann-A We did upgrade Git, but that happened quite a while ago already. Also, given that we shell out to gzip et al explicitly instead of relying on git-archive(1) to compress the archive for us we are not impacted by the same issue as GitHub.
So this here must have a different root cause. And the fact that the root directory has seemingly changed is the best lead we have:
It seems, that the hash of the release archives has changed due to the rename of the root directory in the tar archive, it has changed from previous
amx-cli-v0.2.22
to currentamx-cli-18c40c6e7e8b5998da52bcf56dbfe549594b76a7
.
- Developer
I've dug into the Gitaly code in this context and iterated on our tests, but given that the root cause seems to be a change in the root directory's name I think this is in fact not an issue in groupgitaly. Something on the calling-side in Rails or Workhorse must have changed.
@igor.drozdov Do you have any idea what could've caused this regression?
- Patrick Steinhardt added groupsource code label and removed groupgitaly label
added groupsource code label and removed groupgitaly label
- 🤖 GitLab Bot 🤖 added devopscreate sectiondev labels and removed devopssystems sectioncore platform labels
added devopscreate sectiondev labels and removed devopssystems sectioncore platform labels
- Developer
Thread from Slack.
Collapse replies - Author
@sean_carroll Thanks, I've just no clue how to read that :) Is it public? Can we get some short summary here?
- Developer
Thanks @ynezz that's internal to GitLab.
- Maintainer
Hi @ynezz!
I took a look on this problem, but I cannot reproduce it. When I download the archive from the link you provided it has a correct checksum3faa2d50190524baa59f73d2af322d8c6a27be882d8d2bddd2e80384d8b93eb0
.Does this problem still occur for you? If yes, can you please provide reproduction steps? You also mentioned build failures, can you share link to failed pipelines if they are public?
Collapse replies - Author
Does this problem still occur for you?
Yes.
If yes, can you please provide reproduction steps?
$ curl -v https://gitlab.com/prpl-foundation/components/ambiorix/applications/amx-cli/-/archive/v0.2.22/amx-cli-v0.2.22.tar.gz | sha256sum % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 172.65.251.78... * TCP_NODELAY set * Connected to gitlab.com (172.65.251.78) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS Unknown, Certificate Status (22): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Unknown (8): { [19 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [2412 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [79 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [52 bytes data] * TLSv1.3 (OUT), TLS change cipher, Client hello (1): } [1 bytes data] * TLSv1.3 (OUT), TLS Unknown, Certificate Status (22): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [52 bytes data] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=gitlab.com * start date: Mar 1 00:00:00 2023 GMT * expire date: May 30 23:59:59 2023 GMT * subjectAltName: host "gitlab.com" matched cert's "gitlab.com" * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 } [5 bytes data] * TLSv1.3 (OUT), TLS Unknown, Unknown (23): } [1 bytes data] * TLSv1.3 (OUT), TLS Unknown, Unknown (23): } [1 bytes data] * TLSv1.3 (OUT), TLS Unknown, Unknown (23): } [1 bytes data] * Using Stream ID: 1 (easy handle 0x56417e42b640) } [5 bytes data] * TLSv1.3 (OUT), TLS Unknown, Unknown (23): } [1 bytes data] > GET /prpl-foundation/components/ambiorix/applications/amx-cli/-/archive/v0.2.22/amx-cli-v0.2.22.tar.gz HTTP/2 > Host: gitlab.com > User-Agent: curl/7.58.0 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS Unknown, Certificate Status (22): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [230 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [230 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * Connection state changed (MAX_CONCURRENT_STREAMS updated)! } [5 bytes data] * TLSv1.3 (OUT), TLS Unknown, Unknown (23): } [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] < HTTP/2 200 < date: Tue, 28 Mar 2023 11:15:18 GMT < content-type: application/octet-stream < cache-control: max-age=60, public, must-revalidate, stale-while-revalidate=60, stale-if-error=300, s-maxage=60 < content-disposition: attachment; filename="amx-cli-18c40c6e7e8b5998da52bcf56dbfe549594b76a7.tar.gz" < content-security-policy: base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://*.zuora.com/apps/PublicHostedPageLite.do https://gitlab.com/admin/ https://gitlab.com/assets/ https://gitlab.com/-/speedscope/index.html https://gitlab.com/-/sandbox/ https://gitlab.com/assets/ blob: data:; connect-src 'self' https://gitlab.com wss://gitlab.com https://sentry.gitlab.net https://new-sentry.gitlab.net https://customers.gitlab.com https://snowplow.trx.gitlab.net https://sourcegraph.com snowplow.trx.gitlab.net; default-src 'self'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-cloudresourcemanager.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://*.codesandbox.io https://customers.gitlab.com https://*.zuora.com/apps/PublicHostedPageLite.do; img-src * data: blob:; manifest-src 'self'; media-src 'self' data: http: https:; object-src 'none'; report-uri https://sentry.gitlab.net/api/105/security/?sentry_key=a42ea3adc19140d9a6424906e12fba86; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://www.recaptcha.net/ https://apis.google.com https://*.zuora.com/apps/PublicHostedPageLite.do 'nonce-Q22iV2sFoNDmOfZCBNgNaQ=='; style-src 'self' 'unsafe-inline'; worker-src https://gitlab.com blob: data:; form-action 'self' https: http: http: < content-transfer-encoding: binary < etag: "df4e9097f2cddf728da096fb4a0f9dc8" < permissions-policy: interest-cohort=() < referrer-policy: strict-origin-when-cross-origin < x-content-type-options: nosniff < x-download-options: noopen < x-frame-options: SAMEORIGIN < x-permitted-cross-domain-policies: none < x-request-id: 01GWJ7R6X3EA17CRD8KZWS2SN8 < x-runtime: 0.146840 < x-ua-compatible: IE=edge < x-xss-protection: 1; mode=block < gitlab-lb: fe-37-lb-gprd < gitlab-sv: web-gke-us-east1-c < cf-cache-status: HIT < age: 40 < report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Fs%2FWVOqSY0C0yw%2F3qrGHVgbC4%2B0g7ak%2Bi1Tc4cGVN9zFupY2PxtBCUK1SSlGAGz7DQp5a6G0HmSmt3n%2FxjWAmehn8uhYGbZmPPdDHpiKafbpuRxV4uw8wKlO33M%3D"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800} < strict-transport-security: max-age=31536000 < set-cookie: _cfuvid=8I71wOgbavZzgvUcMiFCFqSJfYXLJNNPGwCRCEEOtH4-1680002118995-0-604800000; path=/; domain=.gitlab.com; HttpOnly; Secure; SameSite=None < server: cloudflare < cf-ray: 7aef745baf03b373-PRG < { [486 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] * TLSv1.3 (IN), TLS Unknown, Unknown (23): { [1 bytes data] 100 54512 0 54512 0 0 466k 0 --:--:-- --:--:-- --:--:-- 466k * Connection #0 to host gitlab.com left intact 6f8b753736e6badbcecd11952c6e6691934ebd4fda1f598dfad91855dbf6148f -
You also mentioned build failures, can you share link to failed pipelines if they are public?
This job for example https://gitlab.com/prpl-foundation/prplos/prplos/-/jobs/4010180045
Edited by Petr Štetiar 1 - Author
https://prpl-foundation.gitlab.io/-/prplos/prplos/-/jobs/4010180045/artifacts/logs/package/feeds/feed_prpl/amx-cli/download.txt is the log for the failed download of the same release archive:
$ curl -f --connect-timeout 20 --retry 5 --location --insecure https://gitlab.com/prpl-foundation/components/ambiorix/applications/amx-cli/-/archive/v0.2.22/amx-cli-v0.2.22.tar.gz ... Hash of the downloaded file does not match (file: 6f8b753736e6badbcecd11952c6e6691934ebd4fda1f598dfad91855dbf6148f, requested: 3faa2d50190524baa59f73d2af322d8c6a27be882d8d2bddd2e80384d8b93eb0) - deleting
- Author
@vyaklushin I've just created a VPS and added your https://gitlab.com/vyaklushin.keys there, so you can
ssh root@95.216.152.125
and debug the issue by yourself, it's reproducible there. - Maintainer
Thank you for the additional details @ynezz!
We have discovered the problem and working on the fix.
3 2
- Igor Drozdov assigned to @igor.drozdov and @vyaklushin and unassigned @pks-gitlab
assigned to @igor.drozdov and @vyaklushin and unassigned @pks-gitlab
- Patrick Steinhardt mentioned in merge request gitaly!5580 (merged)
mentioned in merge request gitaly!5580 (merged)
- Maintainer
Setting label(s) Category:Source Code Management based on groupsource code.
- 🤖 GitLab Bot 🤖 added Category:Source Code Management label
added Category:Source Code Management label
- Maintainer
This issue is labeled regression, but doesn't specify which milestone introduced it. We assume it was introduced in the current version (15.11.0-pre) and have labeled it regression:15.11:
- If this version number is wrong, please correct it.
- Keep the regression label. It helps us search for regressions across all versions.
- 🤖 GitLab Bot 🤖 added regression:15.11 label
added regression:15.11 label
Hi. Do you have an ETA on when gitaly!5580 (merged) will land? All our CIs using source-based build systems (and that verify checksums) are failing.
3Collapse replies - Maintainer
Sorry for the inconvenience!
The fix for this problem is ready and it needs a final approval before it's merged. I estimate that it will happen in a few hours. After that, it will be picked up by the deploy process and released to GitLab.com. Depending on how quickly this happens, the problem should be fully resolved today-tomorrow.
4 - Author
Thanks a lot for such a positive update! BTW just out of pure curiosity and since I don't see the fix mentioning this issue, what has caused the problem?
The original sha512 will comeback as it was? Or you will update for new sha512?
1