Split Matrix Processing into "Tiles"
In order to manage larger files, it is possible to split the comparison matrix into smaller "tiles". These tiles can be compared, and the resultsing chains "stitched" together.
The resulting chains can be serialized to disk while waiting for their neighbors to complete processing allowing us to distribute the job across time. That means slower but only in the cases where memory was going to be exceeded anyway.
Also, this would introduce the ability to distribute the job across space (multiple clients).