Refactor for scanner.pas and tokens.pas.

Promoting #39402.

This reduces code repetition in scanner.pas and changes a binary search to a hash table, using a hash function tokens.TokenHash tuned for short strings and existing keywords. My gain is 25% of a newly-introduced recognize_keyword, 1% of tscannerfile.readtoken, or 0.1% of total compilation time.

Replacing custom hash function with slower and generic FPHash will trigger an internalerror safeguard that will complain about too high collision count and suggest you to either weaken itself or to increase hash table size to 2¹¹; even the former case will keep outperforming binary search.

Can conflict with promising changes like #35827.