buildbox-casd crashed after connection error
Context
Please outline the background to the issue. If you are raising a bug, please summarise the bug and include:
- Expected behaviour Buildbox-casd doesn't crash but reports errors through regular protocol
- Current behaviour Buildbox-casd crashed
- Steps to reproduce Unreproducible
- Relevant Log / Screenshot
2022-12-15T21:05:58.905+0000 [189:140368188274240] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 1/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown], retrying in 1000 ms...
2022-12-15T21:05:59.906+0000 [189:140368188274240] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 2/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown], retrying in 1600 ms...
2022-12-15T21:06:01.507+0000 [189:140368188274240] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 3/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown], retrying in 2560 ms...
2022-12-15T21:06:04.068+0000 [189:140368188274240] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 4/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: tcp handshaker shutdown], retrying in 4096 ms...
2022-12-15T21:06:08.165+0000 [189:140368188274240] [buildboxcommon_grpcretrier.cpp:157] [ERROR] Retry limit (4) exceeded for "BatchUpdateBlobs()", last gRPC error was [14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host]
2022-12-15T21:06:08.233+0000 [189:140368188274240] [buildboxcommon_casclient.cpp:599] [ERROR] Batch upload failed: 14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host
2022-12-15T21:06:08.242+0000 [189:140368188274240] [buildboxcasd_localcasinstance.cpp:47] [ERROR] Runtime error uploading and storing path "buildstream-build/subprojects/gst-plugins-base/../..": std::runtime_error exception thrown at [buildboxcasd_localcasproxyinstance.cpp:904], errMsg = "Unable to upload 4 blobs to remote CAS"
2022-12-15T21:06:08.358+0000 [189:140368818980416] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 1/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host], retrying in 1000 ms...
2022-12-15T21:06:09.359+0000 [189:140368818980416] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 2/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host], retrying in 1600 ms...
2022-12-15T21:06:10.960+0000 [189:140368818980416] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 3/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host], retrying in 2560 ms...
2022-12-15T21:06:13.521+0000 [189:140368818980416] [buildboxcommon_grpcretrier.cpp:164] [WARNING] Attempt 4/5 for "BatchUpdateBlobs()" failed with gRPC error [14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host], retrying in 4096 ms...
2022-12-15T21:06:17.618+0000 [189:140368818980416] [buildboxcommon_grpcretrier.cpp:157] [ERROR] Retry limit (4) exceeded for "BatchUpdateBlobs()", last gRPC error was [14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host]
2022-12-15T21:06:17.618+0000 [189:140368818980416] [buildboxcommon_casclient.cpp:599] [ERROR] Batch upload failed: 14: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: No route to host
2022-12-15T21:06:17.618+0000 [189:140368818980416] [buildboxcasd_localcasinstance.cpp:47] [ERROR] std::runtime_error exception thrown at [buildboxcasd_localcasproxyinstance.cpp:943], errMsg = "Unable to upload 1 blobs to remote CAS" thrown for path: /builds/freedesktop-sdk/freedesktop-sdk/cache/buildstream/tmp/tmpy4sk3ek0
2022-12-15T21:06:17.685+0000 [189:140368869859264] [buildboxcasd.m.cpp:124] [INFO] Received signal [15], stopping and exiting...
Task Description
Describe the steps to take / implementation plan. If it's a bug, do you know how it may be tackled?
-
A check-list could be used here.
Acceptance Criteria
Identify how this is to be proven done before the issues can be closed, eg. links to the relevant documentation completed or demo location, whatever it is that is that is being 'delivered' here.
Consider whether the following are required, and complete if so:
-
Unit tests -
Metrics -
Documentation update(s)