I've added arrows on the right side, pointing to the values we were seeking for. Now we know what these numbers means in term of the parser states:
I've added arrows on the right side, pointing to the values we were seeking for. Now we know what these numbers mean in term of the parser states:
-`1`is `Strings`
-`9`is `Operators`
-`11`is `Keywords`
-`1`represents `Strings`
-`9`represents `Operators`
-`11`represents `Keywords`
Let's analyze the plugin output:
Let's analyse the plugin output:
![state-IDs plugin ON][state-IDs ON]
We can now get a clear picture of how HL parser is tokenizing the "`print("Hello!")` line, step by step:
We can now get a clear picture of how HL parser is tokenizing the “`print("Hello!")`” line, step by step:
| token | state ID | parser state |
...
...
@@ -114,6 +114,7 @@ You'll' also notice that `syntaxUpdate()` is being called twice for tokens insid
In complex language definitions, the parser might go through multiple updates to evaluate each token, depending on the token's context and the definitions provided by the syntax, but especially if there are custom rules hooked into `OnStateChange()` that force it to return with custom values (eg: `HL_REJECT`, `HL_STANDARD`, etc.).
Playing around with the [state-IDs plugin] and following the parser's syntax updates and state changes with various input examples and languages — while studying their syntax definition code — is a great way to gain insights on Highlight's internals and how custom code in the hook functions can alter the parser's behaviour.
