Skip to content

Optimize transaction committed log retention

We introduced a mechanism to keep track of committed log entries after it's admitted by the manager. Those logs are used for conflict checks, verification, result merges, etc. They are organized as an in-memory linked list. Empty leading log entries are truncated if there isn't further transaction using them as its snapshot repositories.

That linked list stores the content of the log entry at the moment. Although each entry does not contain heavy data, they might be kept for a significant amount of time. For example, a repacking task might run for hours until being applied. We need to retain all entries from the time the task starts until then. A busy repository can make the list accumulated. Thus, memory is not the optimal place for it.

Our database already contains all the log records. We can offload the entry content from memory and re-use saved log entry content in the database. At this point, a log entry is removed right after it's applied. We need to let the manager remove a log entry only if it's lower than the low-water mark, which is the front of the committed entry list. If an entry is not deleted after applied, it will be deleted when the referring transaction finishes eventually.

Merge request reports