Skip to content

Add missing tests coverage to the s3-aws driver

We are currently missing tests coverage for the following cases:

  • parameter parsing:
    • chunksize
    • multipartcopychunksize
    • multipartcopymaxconcurrency
    • multipartcopythresholdsize
    • maxrequestspersecond
    • maxretries
    • object ACL parameter should not be set when object ownership is enabled
  • getParameterAsInt64:
    • conversion from string, int64, int, []byte,
    • min and max
  • Writer:
    • s3Path(path) != *multi.Key - existing multi-part upload? this is a bigger issue, we should re-think whether we should be resuming multipart uploads at all.
  • Stat:
    • fi.IsDir = true - never reached as we do not set Delimiter field
  • Move:
    • lastByte = fileInfo.Size() - 1 - adjust lastbyte size
  • URLFor
    • method != GET and method != HEAD (Post?)
  • Walk is not covered at all
  • WaltPararell:
    • fallback to Walk because parallel is not supported
    • path == ""
    • empty bucket
  • Writer.Write:
    • old file is bigger than minChunkSize
  • during multipart upload, if the last part is smaller than minpartsize, the file is re-uploaded, but what if the file is larger than 5GiB (i.e. max part size)?
  • during flush, the pending and ready parts both have w.driver.ChunkSize size, and we do Commit() plus Close() - I check if here is going to be a data corruption here
  • use multi-error in copy() func for pararell multi-part upload.
  • resp, err := d.S3.ListObjectsV2WithContext(ctx, listObjectsV2Input) in the Delete() call - it should not treat error equal to resp.Contents == 0
  • Stat call and all other that use ListObjectsV2Input should be precise when checking the output of the list as this only does a prefix match, and it should be matching on up to delimiter for the paths received. I have already added DirPartialPrefix test for the Stat() call
  • chunk size should be smaller than 2x chunk size because of how flushPart() method works - if if int64(len(w.pendingPart)) < w.driver.ChunkSize { to wtedy pending i ready parts are concatenated which could result in double the size (well, almost)
Edited by Pawel Rozlach