Help Understanding AddKeyword() and OnStateChange()

Could you please help me undestand better how AddKeyword() could be used in syntax defitions.

The plugins documentation mentions:

The functions AddKeyword, RemoveKeyword and OnStateChange are also useful in language definitions without a plug-in use case.

Failed Attempt at AddKeyword()

While drafting the HEXAPUNKS syntax, I initially tried to use AddKeyword() the create a list of Labels as they were detected during parsing.

Since the token following a MARK instruction would always be a label, and because labels could also occur after JuMP, TJMP and FJMP (but these might also be followed by a register), I tried this:

Keywords = {
  -- A token following MARK is always a Label:
  { Id = 1,
    Regex = [=[ ^(?i)(mark)\s*(\w+) ]=],
    Group = 2
  }
}

function OnStateChange(oldState, newState, token, kwGroupID, lineno, column)
  if newState==HL_KEYWORD and kwGroupID==1 then
    -- Add the Label found after MARK to the list of Labels:
    AddKeyword(token, 1)
  end
  return newState
end

The idea was that this should allow to reckognize further occurences of that label in JUMP statements.

The above solution didn't work, and I've also tried various tweaks but none worked.

• Would this be possibile to achieve?
• Also, would keywords added via AddKeyword() match occurences before their addition?

Usually labels are used in JUMPs before the actual MARK definition, so for this to work Highlight would need to process the source after having created the new entries via AddKeyword() (but I've tested source files where the Labels also appear after their MARK definition, but they still didn't get highlighted).

Advanced Tips & Tricks

I would like to learn how to exploit AddKeyword() and other Lua hacks to create some smart syntaxes, and document them in the Wiki under "Syntaxes Tips & Tricks".

For example, I'm struggling in my attempt to create a langDef for AsciiDoc, because its syntax is quite intricate, and I'll need to find a way to track states and add kewyords on the fly. Another tricky syntax I've tried to creat is PCRE, where I wanted to use custom Lua variables to track nesting states and contexts.

Using OnStateChange() to Modify Elements

Another thing which isn't clear to me is wether it's possible to use OnStateChange() to change the element of a matched token (e.g. to convert specific tokens listed as keywords to something else).

For example, it would be useful to create a placeholder kwd group (e.g. with Group Id=99) and then use OnStateChange() to evaluate the actual context via state variables and then assign the matched token to a different kwd group (or element) depending on the current contex.

I've experimented with this but I got the impression that the only valid return values are newState, HL_REJECT, and that you can't for example return HL_NUMBER to convert a token from keyword to number if it meets certain criteria.

Hopefully these examples illustrated what I'm trying to achieve.

• Is this even possible?
• Are thre any syntaxes that employ advanced techniques of this type?