Skip to content

Unify logging transaction's operations and files

Sami Hiltunen requested to merge smh-unify-log-committing into master

TransactionManager is currently storing a log entry's state in two locations:

  1. The protobuf message in the database.
  2. The supporting files in a directory on the filesystem.

This is mostly due to a historical reasons. Initially we didn't have mechanism to log files. The log entry was completely self-contained in the protobuf message and it was easier to store it in the database. When we started logging packfiles, we wanted to log them without copying them as they could be very large. They are thus committed by just moving them to a correct location on the filesystem.

Splitting the state makes things more complex than they have to be:

  1. It is more complex, and thus difficult to understand. It's been a topic of confusion already a few times that the log entries are not entirely contained in the same place.
  2. Committing the log changes atomically requires additional work.
    • We need to handle cases where committing the protobuf message works but committing the files didn't.
    • We need to handle cases where removing the protobuf message worked but committing the files didn't.

This commit simplifies this by making the log entry self contained by storing all of its state in the directory on the filesystem. This way we no longer have a separate protobuf message in the database, and we can commit and remove the log entries atomically by moving and removing the log entry's state from the WAL directory. The protobuf message that used to be in the database is now stored as a 'MANIFEST' file in the directory. The file name is chosen as with the upcoming physical logging protocol the log entries will contain the manifest and a collection of files. The manifest describes what operations to perform and where to place these files.

External behavior of transactions remain the same so no changes are needed in the tests. Two tests that are asserting internals of the manager were updated to handle the new way to store the log entries.

Closes #5818 (closed)
Closes #5810 (closed)

Merge request reports