s3 deletes are not retried
A bunch of jobs failed recently:
- https://gitlab.com/gitlab-org/container-registry/-/jobs/11176429272
- https://gitlab.com/gitlab-org/container-registry/-/jobs/11176429274
- https://gitlab.com/gitlab-org/container-registry/-/jobs/11176429266
- https://gitlab.com/gitlab-org/container-registry/-/jobs/11176429270
- https://gitlab.com/gitlab-org/container-registry/-/jobs/11176511551
For example, from https://gitlab.com/gitlab-org/container-registry/-/jobs/11176429272:
=== RUN TestS3DriverSuite/TestConcurrentFileStreams
testsuites.go:3401: blob path for this stream: /8-vw2s4i9zrg.y_lt1-eb_8m/1ig2/m
testsuites.go:3401: blob path for this stream: /vt6qxx2mn_7/oc_6jow0u84k_fwy1o9
testsuites.go:3401: blob path for this stream: /59_kk_o.oji/e.6l5wfj/l-0j71f2/x
testsuites.go:3401: blob path for this stream: /x77rd5v11/ao/l-qjyoh_n9b/4x_i/l
testsuites.go:3401: blob path for this stream: /ovxoqgne-sggqs8u777rkds_229u4.k
testsuites.go:3401: blob path for this stream: /z.ok_gtaomd7m_0.sw.m/am-zyk/jki
testsuites.go:3401: blob path for this stream: /7vr-ln.u_3x-mux5j-x-e_c2id-6/85
testsuites.go:3401: blob path for this stream: /q-kr_9.h2cnvgys-a8v/vd.wra66r/d
testsuites.go:3401: blob path for this stream: /o7f-pk/p_23_ps2.c/50g6_h_bc/z39
testsuites.go:3401: blob path for this stream: /8-pva2xdb_3-ctjya_404_xr.8gq_hz
testsuites.go:3401: blob path for this stream: /4tvbv7cu4e73scl_e_kv/n38d6z3/by
testsuites.go:3401: blob path for this stream: /o-se_1.y/jbgq5_ae38iez.wr.yp19j
testsuites.go:3401: blob path for this stream: /2ra7q.ils0/q4z4b_etcql85-8_4pbc
testsuites.go:3401: blob path for this stream: /it-y/fx/z_1ih-o.2.xga.v/u3j/1/x
testsuites.go:3401: blob path for this stream: /xyn.vx.rlg3pl6wjea6b2oc7ys0-50f
testsuites.go:3401: blob path for this stream: /utr-y17vsf/1tni/akw.h6wq/t/vuvb
testsuites.go:3401: blob path for this stream: /p64l7ixmr4g1d0wo/1/r4h/6-nlz/ya
testsuites.go:3401: blob path for this stream: /3b_z7ux/ge4a.4hvz/c/5/8c.y/5d/f
testsuites.go:3401: blob path for this stream: /hx.eiv/6-y3jp_wa6-0_j-2atmo62/2
testsuites.go:3401: blob path for this stream: /akqoy4sj-y9/k97/7/v/albp.3vc/6t
testsuites.go:3401: blob path for this stream: /k-b-z_8/001g-v.a5f/6i/s_l7_5d/7
testsuites.go:3401: blob path for this stream: /p_wa.ga/f_mz-4.f1.wl/l2izpn.eis
testsuites.go:3401: blob path for this stream: /gd4gwr.4_lyip/zki/5_vn3v/xk/0o3
testsuites.go:3401: blob path for this stream: /39-h.y7aipxxm-qvsb_f/6/ypdx/y67
testsuites.go:3401: blob path for this stream: /qqllh.a_s4.jq_omm.r/z0qivn1lv/g
testsuites.go:3401: blob path for this stream: /8th-t99kq2_z0/3/2we-t8w-6_lo-ts
testsuites.go:3401: blob path for this stream: /7xy_4qnwoz.wp120j1bpppbv.asu_hr
testsuites.go:3401: blob path for this stream: /aw-p0/8hu9ap3gh2_5_mt6-w136p5rv
testsuites.go:3401: blob path for this stream: /83f7g-cu-f5.tjs32i.ou_x.e5-ypyh
testsuites.go:3401: blob path for this stream: /agao5-lmfqcs_ieup.o9/s-vb30/i/h
testsuites.go:3401: blob path for this stream: /cuw6-ce-yj8wac_6x9_xu/7cs4/ef8y
testsuites.go:3401: blob path for this stream: /9m4fap-b_7.nahh8_1_h.xkd/zfz5/z
testutil.go:169:
Error Trace: /builds/gitlab-org/container-[MASKED]/[MASKED]/storage/driver/internal/testutil/testutil.go:169
/builds/gitlab-org/container-[MASKED]/[MASKED]/storage/driver/testsuites/testsuites.go:256
/builds/gitlab-org/container-[MASKED]/[MASKED]/storage/driver/testsuites/testsuites.go:3443
/builds/gitlab-org/container-[MASKED]/[MASKED]/storage/driver/testsuites/testsuites.go:2370
/usr/local/go/src/runtime/asm_amd64.s:1700
Error: Received unexpected error:
s3_v2: 1 error occurred:
* deleting file 'tmp/TestS3DriverSuite3678674193/001/7xy_4qnwoz.wp120j1bpppbv.asu_hr': 'We encountered an internal error. Please try again.'
Test: TestS3DriverSuite/TestConcurrentFileStreams
--- FAIL: TestS3DriverSuite/TestConcurrentFileStreams (2.69s)Basing on the logs, the request was not retried, we need to investigate why:
2025-08-28-22-21-31-14066BD3048CD36D:a7f28db387c055e2274560cd0eXXXXXXXXXXXX573ad6fa17100a2bb4148ea46b mybucket [28/Aug/2025:21:34:20 +0000] XXX.XXX.XXX.XXX arn:aws:REDACTED ZTSDZ624FK4VSZHD REST.PUT.OBJECT tmp/TestS3DriverSuite61638898/001/qfjgyum58q/a8e_ido513r6_r/zd-a_wb7.yh43vl5sdt_n-g5pj.ltd7q71.e9_t-fo7 "PUT /tmp/TestS3DriverSuite61638898/001/qfjgyum58q/a8e_ido513r6_r/zd-a_wb7.yh43vl5sdt_n-g5pj.ltd7q71.e9_t-fo7?x-id=PutObject HTTP/1.1" 200 - - 8 18 13 "-" "aws-sdk-go-v2/1.38.1 ua/2.1 os/linux lang/go#1.23.12 md/GOOS#linux md/GOARCH#amd64 api/s3#1.86.0 docker-distribution-unknown-go1.23.12 m/0,Z" - TTSDkB6bmMb6j1aj11Yxtz3QCGOTq90crZf9cjhT4Y6eVzsrEPPnk5zVxn0xwP9EFCZPiDnVJ7mFmpXxqvANxey0SpJMxyGk SigV4 TLS_AES_128_GCM_SHA256 AuthHeader mybucket.s3.us-east-1.amazonaws.com TLSv1.3 - -
2025-08-28-22-38-04-2B3D5BD91831EB49:a7f28db387c055e2274560cd0eXXXXXXXXXXXX573ad6fa17100a2bb4148ea46b mybucket [28/Aug/2025:21:34:23 +0000] XXX.XXX.XXX.XXX arn:aws:REDACTED ZG9C6WT51KY7M87B BATCH.DELETE.OBJECT tmp/TestS3DriverSuite61638898/001/qfjgyum58q/a8e_ido513r6_r/zd-a_wb7.yh43vl5sdt_n-g5pj.ltd7q71.e9_t-fo7 - 500 InternalError - - - - - - - IaH19n07QxCBWQWdeRXHQYQj/lKaQ27Mg3o0n1uzG7Sy7m0AezHHh3NVmeA2ZjXxYPeAtbMUqjXiCwm96aPYojST77r4myOp SigV4 TLS_AES_128_GCM_SHA256 AuthHeader mybucket.s3.us-east-1.amazonaws.com TLSv1.3 - -
2025-08-28-22-44-41-F060D135F30E9173:a7f28db387c055e2274560cd0eXXXXXXXXXXXX573ad6fa17100a2bb4148ea46b mybucket [28/Aug/2025:21:34:24 +0000] XXX.XXX.XXX.XXX arn:aws:REDACTED CB93962YZKWVBD4F BATCH.DELETE.OBJECT tmp/TestS3DriverSuite61638898/001/qfjgyum58q/a8e_ido513r6_r/zd-a_wb7.yh43vl5sdt_n-g5pj.ltd7q71.e9_t-fo7 - 204 - - 8 - - - - - Su/4B+Vtt0o+K8C6BcEovi43Tq0W8vtW8Hsdl2s6zpWxMwrZaNOLMn+Klhq6nNiSU4CGGHGePJw= SigV4 TLS_AES_128_GCM_SHA256 AuthHeader mybucket.s3.us-east-1.amazonaws.com TLSv1.3 - -Edited by Pawel Rozlach