A Python tool for splitting large Markdown files into smaller sections based on a specified token limit. This is particularly useful for processing large Markdown files with GPT models, as it allows the models to handle the data in manageable chunks.
python nlp markdown natural-language-processing text-analysis openai text-summarization summarization text-processing gpt data-preprocessing mistletoe split-text text-tokenization openai-gpt gpt-3 gpt-4 gpt-35-turbo gpt-35-turbo-16k markdown-processing
-
Updated
May 1, 2025 - Python