-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
Add encode_byte_offsets methods for byte-level alignment
#2049
opened May 2, 2026 by
kashif
Loading…
Add scaling_bench: encode_batch vs worker-pool comparison (#1900)
#2048
opened May 1, 2026 by
stargazerZJ
Loading…
5 of 6 tasks
Cyrillic normalizer and decoder for south slavic languages
#2046
opened Apr 28, 2026 by
procesaur
Loading…
perf(unigram): pre-size token map and replace per-node HashMap with Vec
#2039
opened Apr 26, 2026 by
taeyun16
Loading…
feat(ByteLevel): skip per-byte transform for printable-ASCII tokens
#2038
opened Apr 26, 2026 by
KimYannn
Loading…
2 of 3 tasks
feat(NFC): skip Unicode pass for all-ASCII inputs
#2037
opened Apr 26, 2026 by
KimYannn
Loading…
2 of 3 tasks
feat: SIMD ASCII fast path for Lowercase normalizer (~30-49x)
#2036
opened Apr 26, 2026 by
KimYannn
Loading…
6 of 7 tasks
perf(byte_level): port GPT-2 split regex to logos FSM (−22% on GPT-2 encode)
#2031
opened Apr 23, 2026 by
ArthurZucker
Collaborator
Loading…
Real-world (batch × input_length) tokenizer benchmark + cross-library leaderboard
#2030
opened Apr 23, 2026 by
ArthurZucker
Collaborator
Loading…
4 tasks
perf: skip alignment tracking in encode_fast normalization
#2022
opened Apr 10, 2026 by
ArthurZucker
Collaborator
Loading…
feat: Normalizer::normalize_str — skip NormalizedString allocation
#2020
opened Apr 10, 2026 by
ArthurZucker
Collaborator
Loading…
feat: open-addressing merge table with cache-line-local linear probing for BPE
#2012
opened Apr 8, 2026 by
ArthurZucker
Collaborator
Loading…
feat: compact vocabulary — single-allocation id→token store for BPE
#2011
opened Apr 8, 2026 by
ArthurZucker
Collaborator
Loading…
feat(pattern): parallel regex
find_matches for large inputs
#2003
opened Mar 31, 2026 by
McPatate
Member
Loading…
fix: skip serializing ByteLevel fields at their default value
#2001
opened Mar 30, 2026 by
ArthurZucker
Collaborator
Loading…
feat: performance, adding pcre2 backend + regex-shards (5-15% speedup)
#1968
opened Mar 19, 2026 by
michaelfeil
Contributor
Loading…
Previous Next
ProTip!
Type g p on any issue or pull request to go back to the pull request listing page.