[bst-1] bst-artifact-server fails on blobs that are "too large"

Summary

Trying to push artifacts with blobs which are rather nebulously "too large for the server" causes failures. The client will show an error along the lines of "failed to push $key: 13". The server is shows errors like the following in /bstcache/cas/logs

2019-11-26T12:28:56.021+0000 [6:140640094627584] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmpku39pvr1": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:29:53.196+0000 [6:140640103020288] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmputsle4j2": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:30:54.491+0000 [6:140640069449472] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmpi1vr3f2d": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:31:49.797+0000 [6:140640094627584] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmp6da3q7dt": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:32:45.618+0000 [6:140640094627584] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmpx7mavw6_": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:33:39.168+0000 [6:140640077842176] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmpdvmg34yi": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:46:29.978+0000 [6:140640103020288] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmpfxmc87v5": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:47:27.538+0000 [6:140640094627584] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmp_powz_rk": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:48:25.825+0000 [6:140640069449472] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmp748i9b07": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:49:17.993+0000 [6:140640077842176] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmp9n421wnf": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:50:10.252+0000 [6:140640094627584] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmpnzwey9u0": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)
2019-11-26T12:51:02.873+0000 [6:140640069449472] [buildboxcasd_localcasinstance.cpp:34] [ERROR] Invalid argument error uploading and storing path "/data/tmp/tmpr5tcirwu": Digest b26b2ebe36b69a2fcb42707c27cb96365daea586d55e18dcab0874fa374694c4/611728144 does not match data (expected ee6833481ed8a959ec847d0882ac3378027ecbe57c28f187cd7ad0911a92b6d2/268435456)

The server had 1G of ram, the errors stopped when I bumped to 2G. Not all the failures were on the same amount of data, but all are a nice round number of mebibytes (64, 128, 256).

I suspect buildbox-casd is the issue here really.