Skip to content

Draft: Implement expression lexer

Scanner implementation that supports the following:

  • Numbers:
    • decimals (ascii digits 0-9)
    • floats (0.5, .5)
    • scientific notation (1.5e5)
  • Strings: "double" (escapes: \\, \"), 'single' (escapes: \\, \')
  • Identifiers: unicode "letter" and "digit", cannot start with digit
  • Operators:
    • Logical: ==, !=, &&, ||, >, >=, <, <=, !
    • Arithmetic: +, -, *, /, %
  • Punctuation: ()[]{}:,.
  • A number of reserved keywords that cannot be used as identifiers
  • Ignored whitespace
  • UTF8 input, and token offsets are based on runes, not bytes

Supporting more tokens than our parser will initially support is not a problem, because that will provide it's own error if it encounters something unexpected.

I'm adding these tokens now because it's likely the minimum set required to eventually turn this into a useful language. It also sets the standard for later additions.

Closes #227 (closed)

Edited by Arran Walker

Merge request reports

Loading