Increase retry delay when creating and configuring S3 bucket
Created by: viadanna
Seem AWS IAM slowness is preventing successful provisioning of S3 buckets in OCIM after the upgrade to boto3
.
This PR increases the delay between retries during provisioning to account for propagation delays in IAM.
The integration tests are currently failing with:
28/Feb/2019 17:47:46 | INFO | process=2307 | instance.models.instance | instance=10 (Integration - t) | Provisioning S3 bucket...
28/Feb/2019 17:47:47 | INFO | process=2307 | instance.models.instance | instance=10 (Integration - t) | Retrying bucket creation due to "InvalidAccessKeyId", attempt 1 of 4.
28/Feb/2019 17:47:51 | INFO | process=2307 | instance.models.instance | instance=10 (Integration - t) | Retrying bucket creation due to "InvalidAccessKeyId", attempt 2 of 4.
28/Feb/2019 17:47:55 | INFO | process=2307 | instance.models.instance | instance=10 (Integration - t) | Retrying bucket creation due to "AccessDenied", attempt 3 of 4.
28/Feb/2019 17:47:59 | INFO | process=2307 | instance.models.instance | instance=10 (Integration - t) | Retrying bucket creation due to "AccessDenied", attempt 4 of 4.
28/Feb/2019 17:47:59 | CRITICAL | process=2307 | instance.models.instance | instance=10 (Integration - t) | Traceback (most recent call last):
File "/home/circleci/project/instance/logging.py", line 51, in wrapper
return method(self, *args, **kwds)
File "/home/circleci/project/instance/models/openedx_instance.py", line 273, in spawn_appserver
app_server = self._spawn_appserver()
File "/home/circleci/project/instance/models/openedx_instance.py", line 247, in _spawn_appserver
self.provision_s3()
File "/home/circleci/project/instance/models/mixins/storage.py", line 430, in provision_s3
self._create_bucket(location=self.s3_region)
File "/home/circleci/project/instance/models/mixins/storage.py", line 309, in _create_bucket
self._perform_create_bucket(location_constraint)
File "/home/circleci/project/instance/models/mixins/storage.py", line 394, in _perform_create_bucket
bucket = self.s3.create_bucket(Bucket=self.s3_bucket_name)
File "/home/circleci/project/venv/lib/python3.5/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/circleci/project/venv/lib/python3.5/site-packages/botocore/client.py", line 624, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
With this change, integration successfully provisions:
28/Feb/2019 19:18:25 | INFO | process=2286 | instance.models.instance | instance=10 (Integration - t) | Provisioning S3 bucket...
28/Feb/2019 19:18:25 | INFO | process=2286 | instance.models.instance | instance=10 (Integration - t) | Retrying bucket creation due to "InvalidAccessKeyId", attempt 1 of 4.
28/Feb/2019 19:18:40 | INFO | process=2286 | instance.models.instance | instance=10 (Integration - t) | Successfully created S3 bucket.
28/Feb/2019 19:18:41 | INFO | process=2286 | instance.models.instance | instance=10 (Integration - t) | Successfully updated bucket policies.
28/Feb/2019 19:18:41 | INFO | process=2286 | instance.models.instance | instance=10 (Integration - t) | Provisioning RabbitMQ vhost...