Implement Inko's lexer in Inko
Summary
The Ruby compiler's lexer (also written in Ruby) should be ported to Inko, in the module std::compiler::lexer
.
Motivation
Inko's compiler will be gradually ported over from the current Ruby compiler to an Inko based compiler. The first step in this process is to port the lexer, followed by porting over the parser. The lexer will be available as part of the standard library, allowing future tooling to make use of it. For the foreseeable future the API will remain unstable, and as such might change at any point in time. As we get closes to 1.0.0 we will decide what will be part of the public API.
Implementation
For the most part we'd just port the Ruby lexer to Inko, adjusting it to better fit Inko as a language where necessary.
Drawbacks
The Inko lexer will likely be slower than the Ruby lexer, due to the complete lack of optimisations performed by Inko's compiler at the moment. This however is an opportunity for measuring Inko's performance and improving it in the future.