Implement Inko's lexer in Inko itself
Details
This adds std::compiler::lexer, which can be used to lex Inko source code into a sequence of tokens. These tokens in turn can be used to build an Abstract Syntax Tree.
Corresponding issue
Checklist
-
Added tests -
Added documentation -
Inko source code follows the Inko style guide
Edited by Yorick Peterse