Draft: Implement expression lexer
Scanner implementation that supports the following:
- Numbers:
- decimals (ascii digits 0-9)
- floats (0.5, .5)
- scientific notation (1.5e5)
- Strings: "double" (escapes:
\\
,\"
), 'single' (escapes:\\
,\'
) - Identifiers: unicode "letter" and "digit", cannot start with digit
- Operators:
- Logical:
==
,!=
,&&
,||
,>
,>=
,<
,<=
,!
- Arithmetic:
+
,-
,*
,/
,%
- Logical:
- Punctuation:
()[]{}:,.
- A number of reserved keywords that cannot be used as identifiers
- Ignored whitespace
- UTF8 input, and token offsets are based on runes, not bytes
Supporting more tokens than our parser will initially support is not a problem, because that will provide it's own error if it encounters something unexpected.
I'm adding these tokens now because it's likely the minimum set required to eventually turn this into a useful language. It also sets the standard for later additions.
Closes #227 (closed)
Edited by Arran Walker