Skip to content

Partially resolves "Add GCS and S3 support for task runner s3cmd"

The task-runner is currently tied to minio. #500 (closed) suggest improvements to use S3 and GCS. I can't help on the GCS front, but I as my team is using S3 I will attempt to make the necessary changes to support it. In doing this I am making the reference to S3 compatible object storage more generic instead of tied explicitly to minio and adding additional values to global to identify the S3 connection parameters.

Task-runner uses s3cmd instead of a ruby library to access S3, so I mirrored the parameter names on the gitlab-runner values as opposed to the lfs/artifacts/uploads S3 connection values as the gitlab-runner was much closer in how it was asking for those values.

This does not address others aspects of the task-runner such as restore. An separate merge request will be forthcoming for that functionality.

Finally, while attempting to test these changes with minio (I did not want to break the existing minio integration) I noticed that s3cmd was having a great deal of difficulty making consistent connections to minio. This may have been localized to my AWS, but I was only able to randomly complete backups using minio about half the time.

This partially resolves #500 (closed).

Merge request reports