metal3 / checksum errors on OS image

I ran into the following issue on a capm3 kubadm ubuntu job in the context of MR !2961 (merged).

Some nodes could not come up, and I see the following in console log (bootstrap-cluster-dump/default pod logs):

2024-11-13 13:50:20 ::  [  112.132767] ironic-python-agent[866]: 2024-11-13 13:50:19.642 866 ERROR ironic_python_agent.extensions.standby [-] 
Image failed to verify against checksum. location: /dev/vda; 
image ID: ubuntu-jammy-plain-kubeadm-1-29-9.raw; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: cb40043bb00b59e093bf963ab8b17b8040a31eda34987ef820a31c8d62835a24

The error is repeated, but the verification checksum is always different:

2024-11-13 13:49:44 ::  [   75.583519] ironic-python-agent[866]: 2024-11-13 13:49:43.093 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 3 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: f97c292470cff7001b55d138bb35615d1054279c0b8b473a61845552927c78c5: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: f97c292470cff7001b55d138bb35615d1054279c0b8b473a61845552927c78c5
2024-11-13 13:49:56 ::  [   87.767554] ironic-python-agent[866]: 2024-11-13 13:49:55.278 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 4 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: ec71b0f116d77a7c77e41101b54d2c5b669505c69d44f56d0a1bd3a30f5dcc7c: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: ec71b0f116d77a7c77e41101b54d2c5b669505c69d44f56d0a1bd3a30f5dcc7c
2024-11-13 13:49:56 ::  [   87.767892] ironic-python-agent[866]: 2024-11-13 13:49:55.278 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 4 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: ec71b0f116d77a7c77e41101b54d2c5b669505c69d44f56d0a1bd3a30f5dcc7c: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: ec71b0f116d77a7c77e41101b54d2c5b669505c69d44f56d0a1bd3a30f5dcc7c
2024-11-13 13:50:08 ::  [   99.930750] ironic-python-agent[866]: 2024-11-13 13:50:07.441 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 5 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: 5a39faecd384a13479321dc5aab4523d2509ba4c5a9bdd71912f683328242c2e: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: 5a39faecd384a13479321dc5aab4523d2509ba4c5a9bdd71912f683328242c2e
2024-11-13 13:50:08 ::  [   99.931084] ironic-python-agent[866]: 2024-11-13 13:50:07.441 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 5 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: 5a39faecd384a13479321dc5aab4523d2509ba4c5a9bdd71912f683328242c2e: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda;
 image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: 5a39faecd384a13479321dc5aab4523d2509ba4c5a9bdd71912f683328242c2e
2024-11-13 13:50:20 ::  [  112.132243] ironic-python-agent[866]: 2024-11-13 13:50:19.642 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 6 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: cb40043bb00b59e093bf963ab8b17b8040a31eda34987ef820a31c8d62835a24: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: cb40043bb00b59e093bf963ab8b17b8040a31eda34987ef820a31c8d62835a24
2024-11-13 13:50:20 ::  [  112.132998] ironic-python-agent[866]: 2024-11-13 13:50:19.642 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 6 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: cb40043bb00b59e093bf963ab8b17b8040a31eda34987ef820a31c8d62835a24: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: cb40043bb00b59e093bf963ab8b17b8040a31eda34987ef820a31c8d62835a24
2024-11-13 13:50:33 ::  [  124.455451] ironic-python-agent[866]: 2024-11-13 13:50:31.966 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 7 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: 9266b90282a8b9b01cdc516d02359edde4a7a9f92cde627c35a1e94b71206fd6: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: 9266b90282a8b9b01cdc516d02359edde4a7a9f92cde627c35a1e94b71206fd6
2024-11-13 13:50:33 ::  [  124.455917] ironic-python-agent[866]: 2024-11-13 13:50:31.966 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 7 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: 9266b90282a8b9b01cdc516d02359edde4a7a9f92cde627c35a1e94b71206fd6: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: 9266b90282a8b9b01cdc516d02359edde4a7a9f92cde627c35a1e94b71206fd6
2024-11-13 13:50:45 ::  [  136.612940] ironic-python-agent[866]: 2024-11-13 13:50:44.124 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 8 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: af42660c64afe9305358dd7ecb20fe6085b5d9af0c020b5ade8925144b0ef14b: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: af42660c64afe9305358dd7ecb20fe6085b5d9af0c020b5ade8925144b0ef14b
2024-11-13 13:50:45 ::  [  136.613320] ironic-python-agent[866]: 2024-11-13 13:50:44.124 866 WARNING ironic_python_agent.extensions.standby [-] Image download failed, 8 of 9: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; verification checksum: af42660c64afe9305358dd7ecb20fe6085b5d9af0c020b5ade8925144b0ef14b: ironic_python_agent.errors.ImageChecksumError: Error verifying image checksum: Image failed to verify against checksum. location: ubuntu-jammy-plain-kubeadm-1-29-9.raw; image ID: /dev/vda; 
image checksum: e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c; 
verification checksum: af42660c64afe9305358dd7ecb20fe6085b5d9af0c020b5ade8925144b0ef14b

image checksum is always the same and matches the hash found in os-images-info (coming from OCI artifact annotations): e494bfb3dfd990d691c524ab51fe7dd3318726173e588cf944dc60172f5c230c

So it looks like a file transfer issue.

The os-image-server logs, surprisingly show a transfered bytes count different for each download attempt:

100.72.148.188 - - [13/Nov/2024:13:49:07 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 69404412 "-" "python-requests/2.31.0" "100.72.150.64"
100.72.150.125 - - [13/Nov/2024:13:49:20 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 83463328 "-" "python-requests/2.31.0" "192.168.10.20"
100.72.148.188 - - [13/Nov/2024:13:49:32 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 97515668 "-" "python-requests/2.31.0" "100.72.150.64"
100.72.148.188 - - [13/Nov/2024:13:49:44 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 88806240 "-" "python-requests/2.31.0" "100.72.150.64"
100.72.150.125 - - [13/Nov/2024:13:49:56 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 108611328 "-" "python-requests/2.31.0" "192.168.10.20"
100.72.148.188 - - [13/Nov/2024:13:50:08 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 103604480 "-" "python-requests/2.31.0" "100.72.150.64"
100.72.148.188 - - [13/Nov/2024:13:50:20 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 85665400 "-" "python-requests/2.31.0" "100.72.150.64"
...
100.72.150.125 - - [13/Nov/2024:13:50:33 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 128040768 "-" "python-requests/2.31.0" "192.168.10.20"
100.72.150.125 - - [13/Nov/2024:13:50:45 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 118200000 "-" "python-requests/2.31.0" "192.168.10.20"
100.72.150.125 - - [13/Nov/2024:13:50:57 +0000] "GET /ubuntu-jammy-plain-kubeadm-1-29-9.raw HTTP/1.1" 200 116835328 "-" "python-requests/2.31.0" "192.168.10.20"