Skip to content

speed up backups

Lars Wirzenius requested to merge liw/db-refactor into main

Sponsored-by: author

Rework the way the SQLite database is used, for speed and clarity. I found that the code I originally wrote to abstract SQL use was now quite confusing, error prone, hard to follow, and even harder to confidently modify. The new code consumes and produces backups that are fully compatible with existing releases of Obnam.

Two main changes affect the speed of backups: using a single transaction for the whole backup run, and using cached prepared SQL statements.

For my own use, a no-change backup of my email Maildirs, about 40k files, went from 250 seconds to 5, and most of that time is now download/upload of the SQLite database. Benchmarks with the usual Obnam benchmark suite also show satisfactory improvement.

Also, use Rust 2021 edition to allow newer niceties. Next Debian stable release will have a rustc that supports 2021, so there's no reason for Obnam to not use it.

I will merge this on Monday, unless there are ongoing discussions.

Merge request reports