Skip to content

Increase retry delay when creating and configuring S3 bucket

Boros Gábor requested to merge paulo/increase-create-bucket-delay into master

Created by: viadanna

Seem AWS IAM slowness is preventing successful provisioning of S3 buckets in OCIM after the upgrade to boto3.

This PR increases the delay between retries during provisioning to account for propagation delays in IAM.

The integration tests are currently failing with:

28/Feb/2019 17:47:46 |     INFO | process=2307  | instance.models.instance  | instance=10 (Integration - t) | Provisioning S3 bucket...
28/Feb/2019 17:47:47 |     INFO | process=2307  | instance.models.instance  | instance=10 (Integration - t) | Retrying bucket creation due to "InvalidAccessKeyId", attempt 1 of 4.
28/Feb/2019 17:47:51 |     INFO | process=2307  | instance.models.instance  | instance=10 (Integration - t) | Retrying bucket creation due to "InvalidAccessKeyId", attempt 2 of 4.
28/Feb/2019 17:47:55 |     INFO | process=2307  | instance.models.instance  | instance=10 (Integration - t) | Retrying bucket creation due to "AccessDenied", attempt 3 of 4.
28/Feb/2019 17:47:59 |     INFO | process=2307  | instance.models.instance  | instance=10 (Integration - t) | Retrying bucket creation due to "AccessDenied", attempt 4 of 4.
28/Feb/2019 17:47:59 | CRITICAL | process=2307  | instance.models.instance  | instance=10 (Integration - t) | Traceback (most recent call last):
  File "/home/circleci/project/instance/logging.py", line 51, in wrapper
    return method(self, *args, **kwds)
  File "/home/circleci/project/instance/models/openedx_instance.py", line 273, in spawn_appserver
    app_server = self._spawn_appserver()
  File "/home/circleci/project/instance/models/openedx_instance.py", line 247, in _spawn_appserver
    self.provision_s3()
  File "/home/circleci/project/instance/models/mixins/storage.py", line 430, in provision_s3
    self._create_bucket(location=self.s3_region)
  File "/home/circleci/project/instance/models/mixins/storage.py", line 309, in _create_bucket
    self._perform_create_bucket(location_constraint)
  File "/home/circleci/project/instance/models/mixins/storage.py", line 394, in _perform_create_bucket
    bucket = self.s3.create_bucket(Bucket=self.s3_bucket_name)
  File "/home/circleci/project/venv/lib/python3.5/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/circleci/project/venv/lib/python3.5/site-packages/botocore/client.py", line 624, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied

With this change, integration successfully provisions:

28/Feb/2019 19:18:25 |     INFO | process=2286  | instance.models.instance  | instance=10 (Integration - t) | Provisioning S3 bucket...
28/Feb/2019 19:18:25 |     INFO | process=2286  | instance.models.instance  | instance=10 (Integration - t) | Retrying bucket creation due to "InvalidAccessKeyId", attempt 1 of 4.
28/Feb/2019 19:18:40 |     INFO | process=2286  | instance.models.instance  | instance=10 (Integration - t) | Successfully created S3 bucket.
28/Feb/2019 19:18:41 |     INFO | process=2286  | instance.models.instance  | instance=10 (Integration - t) | Successfully updated bucket policies.
28/Feb/2019 19:18:41 |     INFO | process=2286  | instance.models.instance  | instance=10 (Integration - t) | Provisioning RabbitMQ vhost...

Merge request reports