Replace links to Highlight repository with new URL at GitLab.
General cleanup of Wiki pages: substitute curly quotes with straight quotes; fix TOC of some
pages with new MarkdownTOC system; cleanup lists indentation.
@@ -19,6 +19,7 @@ Useful references on how to debug syntax definition files to pinpoint issue.
...
@@ -19,6 +19,7 @@ Useful references on how to debug syntax definition files to pinpoint issue.
-----
-----
# Introduction
# Introduction
During the creation of a new language definition unexpected parser behaviors might show up in some edge cases. Tracking down the problem is not always easy since Highlight parser is an opaque blackbox to the user — except for its internal state variables which expose the current state of the parser inside hooked functions.
During the creation of a new language definition unexpected parser behaviors might show up in some edge cases. Tracking down the problem is not always easy since Highlight parser is an opaque blackbox to the user — except for its internal state variables which expose the current state of the parser inside hooked functions.
...
@@ -98,20 +99,20 @@ Let's analyse the plugin output:
...
@@ -98,20 +99,20 @@ Let's analyse the plugin output:
![state-IDs plugin ON][state-IDs ON]
![state-IDs plugin ON][state-IDs ON]
We can now get a clear picture of how HL parser is tokenizing the “`print("Hello!")`” line, step by step:
We can now get a clear picture of how HL parser is tokenizing the "`print("Hello!")`" line, step by step:
You'll' also notice that `syntaxUpdate()` is being called twice for tokens inside the string (ie, for “`Hello`” and “`!`”). This means that for the current syntax definition the parser needs to undergo two state updates for evaluating those tokens — basically, one update to establish they are not sub-elements (eg: an escape sequence), and another to establish that the string state needs to carry on.
You'll' also notice that `syntaxUpdate()` is being called twice for tokens inside the string (ie, for "`Hello`" and "`!`"). This means that for the current syntax definition the parser needs to undergo two state updates for evaluating those tokens — basically, one update to establish they are not sub-elements (eg: an escape sequence), and another to establish that the string state needs to carry on.
In complex language definitions, the parser might go through multiple updates to evaluate each token, depending on the token's context and the definitions provided by the syntax, but especially if there are custom rules hooked into `OnStateChange()` that force it to return with custom values (eg: `HL_REJECT`, `HL_STANDARD`, etc.).
In complex language definitions, the parser might go through multiple updates to evaluate each token, depending on the token's context and the definitions provided by the syntax, but especially if there are custom rules hooked into `OnStateChange()` that force it to return with custom values (eg: `HL_REJECT`, `HL_STANDARD`, etc.).