Refactor storage drivers concurrency regulator
Problem
The existing storage driver concurrency regulator (storage.driver.regulator
), which is only used by the GCS and filesystem drivers, has the following main limitations:
- The same concurrency limit applies to all operations, both reads and writes. This is sub-optimal, as blob storage services usually impose different restrictions for reads and write;
- It's not possible to manipulate the regulator from within the storage driver. This means that we cannot implement custom logic (like an increased limit) for a specific driver/method.
Solution
We should refactor the regulator and make it highly configurable. Ideally, it should be then applied to all storage drivers, with different configurations per driver and operation, depending on the underlying storage system capabilities.