Make database backend swappable
There has been a lot of criticism from the start of Conduit using sled as the storage backend. The main arguments against it were:
- It's new and was not tested much yet, making it potentially unreliable.
- There are not many tools to inspect or manipulate the db in case of problems.
We still chose sled because:
- The sled v1.0 release was originally planned i n January. This gave a sense of reliability.
- Sled fits the keep-it-simple architecture Conduit is going for.
- Conduit is very fast with sled, mostly because it's an in-memory database. See romeo juliet benchmark.
However, over time we noticed more problems with sled:
- Memory usage sometimes goes very high with large databases.
- Sled currently deadlocks in some situations
These issues with sled will probably improve at some point (the deadlock is already fixed on sled's master branch), but they left a bad impression.
How should we move forward? I think it is a good idea to invest time into a swappable database backend, so it's easy to change when issues pop up.
What alternatives to sled would you recommend that fit to Conduit? Post them as comments with a short explanation and pros and cons (or link to a page that does so).