Small blocksize causing poor performance on local media

I have:

  • ([x] when completed)
  • searched https://gitlab.com/duplicity/duplicity/-/issues for similar issues. If you find a similar issue and the issue is still open, add a comment to the existing issue instead of opening a new one. If you find a Closed issue that seems like it is the same thing that you're experiencing, open a new issue and include a link to the original issue in the body of your new one.
  • tested that this issue still occurs on the latest stable snap (install instructions: https://snapcraft.io/duplicity), please include the snap version (snap info duplicity | grep installed) output: installed: x.xx.xx (xx)
  • ideally, tested that this issue still occurs on the latest edge snap, if you can test without risking your data. Please include the snap version output: installed: x.xx.xx (xx)

Summary

I recently started using duplicity for backing up to local media (specifically BD-RW and USB flash memory), and noticed the performance was much lower than expected.

For the BD-RW, due to the low blocksize, the performance on reading is also very poor, but for the USB flash memory, only the writing takes a long time.

As an experiment, I increased the block size in this file to 1024*1024: https://gitlab.com/duplicity/duplicity/-/blob/main/duplicity/path.py#L55

And I got much better performance.

Environment

Arch OS

(duplicity --version) duplicity 1.2.2

(command line used)

/usr/bin/duplicity full --progress \
    --include <redacted> \
    --include <redacted> \
    --include <redacted> \
    --include <redacted> \
    --exclude <redacted> \
    --exclude "**" "$HOME"/ \
    --encrypt-key 4FDF72DBB4139F8607173148D77536CA9A841B40 \
    --encrypt-key 84F3F0628144599D9DD42CDAD5F90BD635F4E746 \
    --encrypt-key 52A7E00CE2E6C8041530D8DA7D745F0E8BD88C82 \
    file:///mnt/disc

Steps to reproduce

I think running any larger backup to file:/// directly to optical media will show this behavior.

What is the current bug behaviour?

Small blocksize causing a series of short writes instead of larger writes

What is the expected correct behaviour?

Larger blocksize would be better.

Possible fixes

Several fixes are possible, I don't know which is most acceptable.

  • simply increase block size statically
  • user-adjustable block size (I would prefer this so I can adjust depending on system/target)
  • rework the file copy method to be able to read and write faster. cp command outperforms the function even with larger block size. Maybe I didn't set it large enough? I wouldn't know where to start.

If it is me doing something wrong, corrections are welcome.