LangDefs-Debugging.md

+    Highlight v3.41
+
+# Debugging Language Definitions
+
+Useful references on how to debug syntax definition files to pinpoint issue.
+
+
+-----
+
+**Table of Contents**
+
+<!-- MarkdownTOC autolink="true" bracket="round" autoanchor="false" lowercase="true" lowercase_only_ascii="true" uri_encoding="true" depth="3" -->
+
+- [Introduction](#introduction)
+- [State IDs Plugin](#state-ids-plugin)
+
+<!-- /MarkdownTOC -->
+
+-----
+
+# Introduction
+
+During the creation of a new language definition unexpected parser behaviors might show up in some edge cases. Tracking down the problem is not always easy since Highlight parser is an opaque blackbox to the user — except for its internal state variables which expose the current state of the parser inside hooked functions.
+
+Here you'll find some guidelines and tools on how to leverage those internal states to isolate the problem.
+
+# State IDs Plugin
+
+Highlight ships with [`token_add_state_ids.lua`][state-IDs plugin], a plugin which exposes in the output document the parser's state changes, and their IDs:
+
+``` lua
+Description="Add internal state IDs behind each token (for debugging)."
+
+function syntaxUpdate(desc)
+  function Decorate(token, state)
+    return token .. ' ('.. string.format("%d",state) .. ')'
+  end
+end
+
+Plugins={
+
+  { Type="lang", Chunk=syntaxUpdate },
+
+}
+```
+
+The plugin can be turned ON and OFF in Highlight GUI, from the "Plug-in" tab, allowing to visually track the parser states for the current input file.
+
+A python example, without the State IDs plugin:
+
+![state-IDs plugin OFF][state-IDs OFF]
+
+… and with the State IDs plugin enabled:
+
+![state-IDs plugin ON][state-IDs ON]
+
+
+The plugin adds, at each parser state change, the integer of the new state enclosed in parenthesis. This reveals us interesting details about the parser's inner workings; for example, from the above screenshot we can notice that during the string parsing the parser updates the syntax multiple times, even though the same state is confirmed. This shows us that the parser is consuming the string in chunks, trying to isolate any tokens that could match legitimate sub-string elements (escape sequences, interpolations).
+
+As for the actual numbers, these represent the various possible parser states, which are assigned at initialization time, and might vary with each syntax (depending on what elements are actually defined). The correspondence between parser states and integer values can be retrived via the `--verbose` option on the command line (actual output cut-down here, for space reasons):
+
+```
+> highlight --verbose StatesIDs-plugin-Example.py
+
+Loading language definition:
+C:\Program Files\Highlight\langDefs\python.lang
+
+Description: Python
+
+LUA GLOBALS:
+...
+HL_INTERPOLATION: number [ 10 ]
+HL_INTERPOLATION_END: number [ 19 ]
+HL_KEYWORD: number [ 11 ]                   <-- (11) = Keywords
+HL_KEYWORD_END: number [ 20 ]
+...
+HL_NUMBER: number [ 2 ]
+HL_OPERATOR: number [ 9 ]                   <--  (9) = Operators
+HL_OPERATOR_END: number [ 18 ]
+...
+HL_STANDARD: number [ 0 ]
+HL_STRING: number [ 1 ]                     <--  (1) = String
+HL_STRING_END: number [ 12 ]
+HL_UNKNOWN: number [ 100 ]
+...
+```
+
+
+
+
+[state-IDs plugin]: https://github.com/andre-simon/highlight/blob/master/plugins/token_add_state_ids.lua "View source of 'token_add_state_ids.lua' plugin"
+
+[state-IDs OFF]: ./StatesIDs-plugin-Example_OFF.png "Example of State IDs plugin usage in Highlight GUI (plugin disabled)"
+[state-IDs ON]: ./StatesIDs-plugin-Example_ON.png "Example of State IDs plugin usage in Highlight GUI (plugin enabled)"