Wow @tan_stack Start has server components now! And without « use client »/« use server » It happens within a serverFn so the code explicitly split the server only stuff, no directives needed! I much prefer this approach 🔥

I just released Kiru: a text chunking lib, 1000x faster than LangChain splitters. Built in Rust for Python. If you're building RAG systems, you know the basic drill: load documents, split them into chunks, feed them to your vector database. Simple enough, except most chunking libraries are painfully slow when you're processing millions of documents. kiru solves this bringing Rust native performance to Python via PyO3 bindings. The numbers: - 4000+ MB/s for byte chunking (Pure Rust) - 1,400+ MB/s for byte chunking (Python bindings) - 300+ MB/s for character-aware chunking - Constant memory usage, even on gigabyte files - Works with files, URLs, and glob patterns out of the box Why it's fast and memory efficient: Most chunkers load entire files into memory, then split them. Kiru streams data in blocks, chunks on-the-fly, and never breaks UTF-8 boundaries. The Rust implementation hits 4,000+ MB/s (for in memory string and byte chunking), and even the Python bindings crush slower pure Python implementation. Byte vs Character chunking: Byte chunking splits on byte boundaries (faster, perfect for token-limited models). So for a chunk of 500 bytes we just lookup the 500th byte in the sequence, if it's not a valid character boundary we take the closest one. Character chunking splits on exact characters count. It's slower because you need to compute all the character boundaries (characters can be multi-bytes). Both respect UTF-8 boundaries, so you never get corrupted text. Why it matters: When you're ingesting your entire knowledge base or processing documents in real-time, speed isn't just nice, it's the difference between a system that scales and one that doesn't. What's next: This is just the first release. Coming soon: 👉🏽 Delimiter-based splitting (newlines, sentences, paragraphs) 👉🏽 Recursive splitting with multiple delimiters 👉🏽 Markdown-native chunking (preserve structure) 👉🏽 Sitemap support (chunk entire websites from a single URL) 👉🏽 Code-aware splitting powered by tree-sitter Available now on PyPI and Rust Crates. Fully open source. Links below 👇🏽

😅

Putting it together. Each step is composable and testable.

Every step can fail: - File not found - Invalid YAML frontmatter - MDX compilation error - Schema validation error Effect's type system tracks all these.

Finally, validate against the user's schema. If validation fails, Effect gives us a detailed error with the exact field that's wrong. Type safety from file → database!

Merge frontmatter with compiled output. Now we have everything in one object!

If the user wants exports (like metadata, TOC) we extract them from the MDX content.

For each file, bundle the MDX with mdx-bundler. Effect.tryPromise converts Promise errors to typed errors.