T

tokenizer

Toolchain to translate the Navajo language into American English

Free LLM token counter: see how GPT-4 and GPT-4o split your prompt into tokens, which lines cost the most, and which wordy phrases to cut, with savings measured by re-tokenizing. Runs offline on Linux, macOS and Windows. Uses tiktoken.

Free in-browser token counter: see every token for GPT-5, GPT-4o, GPT-4, Llama 3, Qwen3, DeepSeek, Mistral, rank lines by cost, cut wordy phrases. Nothing uploaded. Live at gpt-token-counter.vercel.app

Rank the lines of a prompt by token cost and fail CI when a prompt goes over its token budget. Offline, uses OpenAI's tiktoken. Python CLI.

GitLab CI/CD components for LLM apps: fail the pipeline on likely hallucinations, prompts over their token budget, padded prompts, an LLM bill heading over budget, or slow and flaky models. Tested on every push.

Utility for doing lexical parsing of some input. It is optimized for static (string) parsing, not partial input stream parsing, but contains tokenizer supporting either.

A privacy-first, institutional-grade KYC framework leveraging Solana Token Extensions (Token-2022) and Zero-Knowledge Proofs. Securely enforce regulatory compliance via Transfer Hooks without exposing sensitive PII on-chain. Designed for DeFi protoco

Use regular expressions to split given string into tokens.

A lightweight document security solution that protects your confidential information when using cloud-based LLMs.

Split texts into words according to spaces and punctuation marks.

Single header source code tokenizer written in ANSI C

Sentence segmenter and tokeniser for Yiddish

chinese-english dictionary based tokenizer for lucene/solr