-
Jamie Tanna authored
The performance of DMD hasn't been particularly great - hence why there's been the need to introduce a progress bar for most of the operations - and it turns out it's to do with SQLite's transactions[0]. As we're not wrapping any of our updates in a transaction, SQLite is producing one for us, which is slower than us managing it ourselves, as we know how many `INSERT`s we'll be doing at a given time. This speeds up considerably, for instance the `integration_test` phase currently takes ~90 seconds instead of ~240 seconds, and results in no longer even seeing the progress bar on `import`s. Closes #89. [0]: https://stackoverflow.com/questions/3852068/sqlite-insert-very-slow
def9666eJamie Tanna authoredThe performance of DMD hasn't been particularly great - hence why there's been the need to introduce a progress bar for most of the operations - and it turns out it's to do with SQLite's transactions[0]. As we're not wrapping any of our updates in a transaction, SQLite is producing one for us, which is slower than us managing it ourselves, as we know how many `INSERT`s we'll be doing at a given time. This speeds up considerably, for instance the `integration_test` phase currently takes ~90 seconds instead of ~240 seconds, and results in no longer even seeing the progress bar on `import`s. Closes #89. [0]: https://stackoverflow.com/questions/3852068/sqlite-insert-very-slow
Loading