Support SPM infill#1492

Merged

abetlen merged 16 commits intoabetlen:mainfrom

CISC:spm_infill

Jun 13, 2024

Commits on May 29, 2024

Support SPM infill
CISC
authored
7bb0b56
Copy full SHA for 7bb0b56
typo--
CISC
authored
c5c056e
Copy full SHA for c5c056e
one less layer of parenthesis necessary
CISC
authored
73a1e72
Copy full SHA for 73a1e72
new required internals
CISC
authored
1f9abf8
Copy full SHA for 1f9abf8
manually add bos/eos if model requires it
CISC
authored
c70483d
Copy full SHA for c70483d
add bos even when unknown
Show description for e54e47e
CISC
authored
e54e47e
Copy full SHA for e54e47e
don't add bos/eos on non-infill pre-tokenized prompt
CISC
authored
fce73d0
Copy full SHA for fce73d0
add tokenizer hack to remove leading space in suffix
CISC
authored
b062762
Copy full SHA for b062762
I keep forgetting metadata are strings
CISC
authored
5118dfa
Copy full SHA for 5118dfa
check if bos exists
CISC
authored
fa97f86
Copy full SHA for fa97f86

Commits on Jun 4, 2024

add example
CISC
authored
7b80669
Copy full SHA for 7b80669

Commits on Jun 5, 2024

Merge branch 'abetlen:main' into spm_infill
CISC
authored
e034e9f
Copy full SHA for e034e9f
add cls/sep instead of bos/eos for WPM vocab
CISC
authored
aab7d32
Copy full SHA for aab7d32
simplify
CISC
authored
2d7cb7e
Copy full SHA for 2d7cb7e

Commits on Jun 12, 2024

color-code filtered suffix
CISC
authored
5a262c6
Copy full SHA for 5a262c6

Commits on Jun 13, 2024

Merge branch 'main' into spm_infill
abetlen
committed
79803b5
Copy full SHA for 79803b5