T

tokenizer

Toolchain to translate the Navajo language into American English

A privacy-first, institutional-grade KYC framework leveraging Solana Token Extensions (Token-2022) and Zero-Knowledge Proofs. Securely enforce regulatory compliance via Transfer Hooks without exposing sensitive PII on-chain. Designed for DeFi protoco

Use regular expressions to split given string into tokens.

Utility for doing lexical parsing of some input. It is optimized for static (string) parsing, not partial input stream parsing, but contains tokenizer supporting either.

A lightweight document security solution that protects your confidential information when using cloud-based LLMs.

Split texts into words according to spaces and punctuation marks.

Single header source code tokenizer written in ANSI C

Sentence segmenter and tokeniser for Yiddish

chinese-english dictionary based tokenizer for lucene/solr