New Bonus Materials: Byte Pair Encoding (BPE) Tokenizer From Scratch #489
Replies: 3 comments · 11 replies
-
|
What I have not yet fully understood is why preserving an empty space in some tokens in the BPE's training process rather treating empty spaces as separate tokens. Is this because it would grow the context window utilization significantly? |
Beta Was this translation helpful? Give feedback.
-
|
Btw for GPT-4, multiple whitespaces have dedicated tokens, which makes it |
Beta Was this translation helpful? Give feedback.
-
|
Interesting - yeah, that makes sense for code indentions! 👍🏻 Actually quite insightful how the tokenizers evolved by time, becoming multimodal 🧠 |
Beta Was this translation helpful? Give feedback.
-
|
BTW Aleph Alpha just proposed a new tokenizer-free autoregressive LLM architecture: |
Beta Was this translation helpful? Give feedback.
-
|
Oh nice! This will be added right to my bookmark list. Kind of reminds me of the byte-latent transformer in December. These tokenizer-free approaches could be a nice article one day. |
Beta Was this translation helpful? Give feedback.
-
Yeah, this would be awesome! |
Beta Was this translation helpful? Give feedback.
-
|
Hah, he's got a good point! Not all zeros are the same kind of zero
…On Sat, Jan 18, 2025 at 6:09 PM Sebastian Raschka ***@***.***> wrote:
Great question. I am also not sure about the history behind it but I
strongly suspect it's because keeping the context window util small. Note
that if you have multiple white spaces after each other, they get treated
as separate white space characters, so white space characters do exist in
GPT-2 tokenizers.
Screenshot.2025-01-18.at.12.08.48.PM.png (view on web)
<https://github.com/user-attachments/assets/5d78d0df-d3d5-403f-b5ef-e3a90e6a9583>
—
Reply to this email directly, view it on GitHub
<#489 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKNJ3VFQJSWQAQLJKGZMJD2LKKF7AVCNFSM6AAAAABVMPI5VWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOBXGY3TONQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***
.com>
|
Beta Was this translation helpful? Give feedback.
-
|
BTW as you mentioned
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
# Initialize tokenizer
tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>"))
# Configure trainer
trainer = BpeTrainer(
vocab_size=1000,
special_tokens=["<|endoftext|>"]
)
# Train the tokenizer
tokenizer.train(files=["the-verdict.txt"], trainer=trainer)
# Save the tokenizer
tokenizer.save("tokenizer.json")
|
Beta Was this translation helpful? Give feedback.
-
Ah yes, this is what @Aananda-giri did in #485 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, exactly. I think it would be great to add this to the introduction text in the notebook at
because HF's |
Beta Was this translation helpful? Give feedback.
-
|
@d-kleine Sure, I can add a note about that. I find HF code really hard to read tbh so I would prefer recommending minBPE. But yeah, I added the note as part of #495 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! 👍🏻 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi all, here's some new bonus material that I thought you might enjoy 😊
Byte Pair Encoding (BPE) Tokenizer From Scratch
Happy weekend!
Beta Was this translation helpful? Give feedback.
All reactions