Skip to content

Fix: Create engine and tokenizer on each call instead of using singletons

Alexander Chueshev requested to merge fix-tokenizer-issue into main

This MR fixes the Already borrowed error when a single tokenizer instance in several threads for truncation - https://github.com/huggingface/tokenizers/issues/537

Ref: #195 (closed)

Edited by Alexander Chueshev

Merge request reports