The new lexer works on the principle of an
incrementing state machine. Since regular
languages are easy to lex, we take advantage
of this to achieve Theta(n) lexing with
small constant factors. Benchmarks show
that the new lexer is significantly faster
than the old lexer, and can lex at speeds
of 1GB/s, as compared to the previous
implementation which only lexed at 25MB/s.