Use pycurl for S3 requests
Context
Currently we use the boto3
library for interacting with S3. It turns out that this library comes with a pretty large amount of CPU overhead, leading to serious GIL contention issues when the S3 CAS backend is under load.
We can improve this situation by moving to a custom client using presigned URLs and pycurl to interact with S3. This will remove a lot of the CPU overhead, and also avoid using Python (and therefore the GIL) for actually sending the HTTP requests, improving the level of GIL contention when under load.