Verified Commit 6c34be9a authored by sunpoet's avatar sunpoet
Browse files

textproc/py-tiktoken: Add py-tiktoken 0.4.0 

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
parent d733ef59
Show whitespace changes
Inline Side-by-side
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment