I just released Kiru: a text chunking lib, 1000x faster than LangChain splitters. Built in Rust for Python. If you're building RAG systems, you know the basic drill: load documents, split them into chunks, feed them to your vector database. Simple enough, except most chunking libraries are painfully slow when you're processing millions of documents. kiru solves this bringing Rust native performance to Python via PyO3 bindings. The numbers: - 4000+ MB/s for byte chunking (Pure Rust) - 1,400+ MB/s for byte chunking (Python bindings) - 300+ MB/s for character-aware chunking - Constant memory usage, even on gigabyte files - Works with files, URLs, and glob patterns out of the box Why it's fast and memory efficient: Most chunkers load entire files into memory, then split them. Kiru streams data in blocks, chunks on-the-fly, and never breaks UTF-8 boundaries. The Rust implementation hits 4,000+ MB/s (for in memory string and byte chunking), and even the Python bindings crush slower pure Python implementation. Byte vs Character chunking: Byte chunking splits on byte boundaries (faster, perfect for token-limited models). So for a chunk of 500 bytes we just lookup the 500th byte in the sequence, if it's not a valid character boundary we take the closest one. Character chunking splits on exact characters count. It's slower because you need to compute all the character boundaries (characters can be multi-bytes). Both respect UTF-8 boundaries, so you never get corrupted text. Why it matters: When you're ingesting your entire knowledge base or processing documents in real-time, speed isn't just nice, it's the difference between a system that scales and one that doesn't. What's next: This is just the first release. Coming soon: 👉🏽 Delimiter-based splitting (newlines, sentences, paragraphs) 👉🏽 Recursive splitting with multiple delimiters 👉🏽 Markdown-native chunking (preserve structure) 👉🏽 Sitemap support (chunk entire websites from a single URL) 👉🏽 Code-aware splitting powered by tree-sitter Available now on PyPI and Rust Crates. Fully open source. Links below 👇🏽

Few days ago I shared FoldCMS. Today I'm showing you how the MDX loader works under the hood. It's actually a great example of @EffectTS_ Streams in action. Let's dive in