Skip to content

Use pycurl for S3 requests

Jürg Billeter requested to merge juerg/pycurl into master

Description

This ports the S3 storage and S3 action cache backends to issue S3 requests using presigned URLs and pycurl, which comes with a lower Python CPU overhead (less GIL contention) and support for parallel requests

Changes proposed in this merge request:

  • Switch moto to server mode for S3 testing
  • Add s3utils module based on pycurl
  • S3Storage: Use pycurl-based s3utils
  • S3Storage: Implement missing_blobs and bulk_*_blobs methods
  • S3ActionCache: Use pycurl-based s3utils

Validation

The existing storage and action cache tests exercise the new code.

Issues addressed

Fixes #386 (closed).

Edited by Jürg Billeter

Merge request reports