Update dependency tokenizers to ^0.19.0
This MR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| tokenizers | dependencies | minor |
^0.15.2 -> ^0.19.0
|
Release Notes
huggingface/tokenizers (tokenizers)
v0.19.1
What's Changed
- add serialization for
ignore_mergesby @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1504
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.19.0...v0.19.1
v0.19.0
What's Changed
- chore: Remove CLI - this was originally intended for local development by @bryantbiggs in https://github.com/huggingface/tokenizers/pull/1442
- [
remove black] And use ruff by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1436 - Bump ip from 2.0.0 to 2.0.1 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1456
- Added ability to inspect a 'Sequence' decoder and the
AddedVocabulary. by @eaplatanios in https://github.com/huggingface/tokenizers/pull/1443 -
🚨 🚨 BREAKING CHANGE🚨 🚨 : (add_prefix_space dropped everything is using prepend_scheme enum instead) Refactor metaspace by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1476 - Add more support for tiktoken based tokenizers by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1493
- PyO3 0.21. by @Narsil in https://github.com/huggingface/tokenizers/pull/1494
- Remove 3.13 (potential undefined behavior.) by @Narsil in https://github.com/huggingface/tokenizers/pull/1497
- Bumping all versions 3 times (ty transformers :) ) by @Narsil in https://github.com/huggingface/tokenizers/pull/1498
- Fixing doc. by @Narsil in https://github.com/huggingface/tokenizers/pull/1499
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.15.2...v0.19.0
Configuration
-
If you want to rebase/retry this MR, check this box
This MR has been generated by Renovate Bot.
Edited by renovate