Edge case affecting transaction Consistency
Final Release Note
Description
By flagging a transaction as "BATCH", an application informs YottaDB that while it requires YottaDB to ensure Atomicity, Consistency and Isolation, it permits YottaDB to relax Durability. A common use case is where a batch process can resume processing from the recovered state of the database, i.e., as long as YottaDB ensures A, C, and I, the application can simply continue from the state the database was recovered to, even if some transactions were not recovered. As part of the commit logic, YottaDB creates journal records in the journal buffers (and the Journal Pool for a replicated instance) but defers the writing of those journal records to the journal file, as well as “hardening” the journal file to persistent storage to subsequent database activity or timers, thereby enhancing application throughput.
In the event a transaction not flagged as BATCH follows a BATCH transaction, its transaction commit ensures Durability of any preceding BATCH transactions, since its Consistency may be based on database state changes made by those BATCH transactions. This Issue documents an edge case in this logic that affects Consistency.
Consider a BATCH transaction T1 that updates regions A, B and C, followed by a transaction T2 that is not flagged as BATCH that updates regions A and C. It reads from B, but does not update B. The commit of T2 should harden the journal files of regions A, B, and C to ensure Consistency, but current database logic only hardens the journal files of regions A and C. This means that if the system crashes after T2 is committed, but before the journal file of B is hardened with the update from T1, a subsequent MUPIP RECOVER/ROLLBACK will flag T1 as a broken transaction, and T2 as a lost transaction because it cannot guarantee the Consistency of T2's updates, even though all of those updates are Durable in the lost transaction file.
Draft Release Note
The edge case of a non-BATCH transaction T2 that follows a BATCH transaction T1 that updates a region read by T2 but not updated by it is fully recoverable. Previously, there was a small interval following such a scenario during which a system crash could result in the subsequent MUPIP RECOVER/ROLLBACK reporting T1 as a broken transaction and T2 as a lost transaction. A complete workaround, potentially with some performance impact, was to not flag transactions as BATCH transactions. On computers with sufficiently performant IO subsystems, the interval could be reduced, but not eliminated, with the SYNC_IO option for journal files. [#655]