Thank you so much, @rasbt ! 🙌

I just checked out your implementation, and it’s amazing! The fact that it’s only 5x faster than Hugging Face implementation is mind-blowing. I’m looking forward to diving deeper into the notebook and understanding the internals.

Your work continues to inspire me, and I’m sure this resource will be incredibly valuable for the entire community. Thanks again for sharing and for tagging me—it’s much appreciated!

The fact that it’s only 5x faster than Hugging Face

"only" 5x faster 😆

Do you any idea why your implementation is way faster than the HF one?

No idea. My guess is it's more simplistic and HF has more bells and whistles because it has to support so many other tokenizers, settings, and so forth. Just my assumption here though.

Just saw that you used HF's GPT2Tokenizer for the comparison - there is also a faster version called GPT2TokenizerFast, see here. Just tested, your implementation is still ca. 2x faster than GPT2TokenizerFast, at least on my hardware.

Thank you so much for the kind words and your interest in the project! 😊

Here are some insights into the tokenizer comparison:

1️⃣ Did the BPE tokenizer lead to more accurate text generation?

Yes, the BPE tokenizer significantly improved tokenization efficiency for the Nepali language by generating fewer and more meaningful tokens compared to the GPT-2 tokenizer.

Example:

• Input text: "राम ले भात खायो ।"
• Tokens generated by GPT-2 tokenizer (tiktoken):
  ['�', '�', 'ा', '�', '�', ' �', '�', '�', '�', ' �', '�', 'ा', '�', '�', ' �', '�', 'ा', '�', '�', '�', '�', ' ', '�', '�']
  (24 tokens, with many fragmented and unusable outputs)
• Tokens generated by BPE tokenizer:
  ['राम</w>', 'ले</w>', 'भात</w>', 'खायो</w>', '।</w>']
  (5 tokens, clean and intuitive for Nepali text)

2️⃣ Performance of BPE tokenizer vs. GPT-2 pretrained tokenizer

• Training: The BPE tokenizer was trained using the Hugging Face Tokenizers library.
• Speed: While the Hugging Face tokenizer provided flexibility and ease of use, it is approximately 10x slower than tiktoken (as noted in this discussion by @rasbt).

3️⃣ More accurate token splitting and generation?

Yes, the BPE tokenizer demonstrated more accurate handling of Nepali morphology and word structures.

Example:

• Input text: "राम घरको छानामाथि बस्यो"
• Tokens generated by BPE tokenizer:
  ['राम</w>', 'घरको</w>', 'छाना', 'माथि</w>', 'बस्यो</w>']

Here, छाना and माथि</w> were correctly split, with माथि</w> recognized as a suffix. This aligns well with the structure of the Nepali language, making the outputs more meaningful.

Improvements and Future Directions

• Handling prefixes/suffixes: While the tokenizer correctly split some suffixes, not all were seperated consistently. Incorporating pre-tokenization techniques (e.g., regex to separate prefixes/suffixes) could further enhance accuracy.
• Rare word handling: Rare words are currently split into individual characters. Increasing the vocabulary size might reduce this issue and improve tokenization for less common words.

I hope this provides a clear overview of the tokenizer comparison and the improvements achieved! Let me know if you’d like further details.

@Aananda-giri I am trying to gauge how much compute required to train tokenizers from scratch. It would helpful to know what percentage of data used from nepali_llm_datasets and what is the peak memory usage? As I am trying something similar.

Unfortunately, I do not have the exact details of the peak memory usage. However, I trained the tokenizer on Google Colab using 12GB of text data. The environment had 12.7GB of RAM and 107GB of storage, which was sufficient for the task.

Later, I trained an updated version of the tokenizer (available here) with approximately 30GB of text data, also on Google Colab.

BTW, what kind of tokenizer are you building (size of your dataset, vocab size)?

Just with Colab notebook, awesome!

I was trying to replicate Gemma like tokenizer, with vocab size 256k, 246GB of Wikipedia dataset with focus on Indic languages.
https://huggingface.co/datasets/ai4bharat/wiki-translate
Here is the gist of the code I used,
https://gist.github.com/karavindhan/63930554f5481d34efdd7f2c5ba2ce56.

Definitely I will checkout minBPE & @rasbt implementaion or try reducing both vocab size and dataset size.