Building applications with LLMs through composability, in Kotlin

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

This repository is part of a course on Elasticsearch in Python. It includes notebooks that demonstrate its usage, along with a YouTube series to guide you through the material.

NeuroRVQ: Multi-Scale Biosignal Tokenization for Generative Foundation Models

Develop DL models using Pytorch and Hugging Face

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Python script for manipulating the existing tokenizer.

Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text

[Unofficial] Simple .NET wrapper of HuggingFace Tokenizers library

Use custom tokenizers in spacy-transformers

Package to align tokens from different tokenizations.

Multi-arch bindings for @huggingface/tokenizers.

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

Python library + CLI for token counting and token distribution analysis in text datasets for LLM data workflows.

DeepSeek-OCR-2-Unlimited-OCR is an advanced, experimental visual document processing and open-ended text localization dashboard. This application establishes a unified interface that allows users to swap between two premier vision-language document models: deepseek-ai/DeepSeek-OCR-2 and baidu/Unlimited-OCR.

High-performance tokenizers

Small library that provides functions to tokenize a string into an array of words with or without punctuation

Visualize some important concepts related to LLM architectures.