Arvind

Reimagined the @Zineps_ai landing page for #ReimagineZineps 🚢 Live: reimagine-zineps.vercel.app One file of hand-written HTML, CSS and JS: an interactive rate engine, an animated "intelligent layer" diagram, a Shipping AI replay and a Workflows canvas. Dark, mint, fast. @iBhanuDahiya call me 📞

35/35 The whole system in one diagram: streaming ingestion, hybrid retrieval, rerank, generation, caches, evals, and tracing. None of it is exotic; all of it is work. Build the boring parts and the demo becomes a product. That's the thread.

34/35 Failure modes to design for: lost-in-the-middle (models skim mid-context), stale index, retrieval drift after re-embedding, prompt noise from over-stuffing, silent empty retrievals. Each has a cheap detector. Add them before users find them.

33/35 Trace every request end to end: spans for rewrite, retrieval (log the chunk IDs!), rerank, and generation. When a user reports a bad answer next week, you replay the exact context that produced it instead of guessing what the index looked like.

32/35 Multi-tenant RAG is a spectrum. Shared index + metadata filter: cheap, one bug from a data breach. Namespace per tenant: the usual balance. Dedicated index per tenant: bulletproof isolation, highest cost. Getting this wrong isn't a bug, it's an incident.

31/35 Freshness is a feature users notice. Wire CDC from your source-of-truth DB into the embedding pipeline and upsert continuously. Then measure staleness (event time to searchable time) as a first-class SLO. "The index is stale" is the real RAG outage.

30/35 Quantization is how big indexes fit in RAM. 100M vectors at 768 dims: float32 307GB, int8 77GB (~zero loss), product quantization ~6GB (recall dip, rescore the top), binary ~10GB (needs rescoring). Compress in RAM, keep full vectors on disk.

29/35 The cost model at scale: embeddings are one-time per doc, index RAM/storage is always-on, rerank and LLM tokens are per-query, and the per-query column dominates. Shrink prompts before you shrink models: context tokens are the bill.

28/35 A p95 RAG request, roughly: query embed 10ms, ANN 20ms, rerank 120ms, LLM generation 2000ms+. Generation dominates everything. So stream tokens, parallelize retrieval, and stop micro-optimizing the vector search: it is not your bottleneck.

27/35 Cache the whole stack. Embedding cache: same text, same vector. Retrieval cache: hot queries skip ANN. Semantic answer cache: a similar-enough question reuses a whole answer. In real traffic the semantic cache absorbs a shocking share of load.

26/35 Scaling the index: shard by tenant or namespace when you can (isolation, easy deletes), by hash when you must. Queries scatter-gather across shards; replicas carry read throughput. Deletes are the hard part of distributed vector search.

25/35 Documents change. Incremental indexing beats weekly full rebuilds: track per-doc content hashes so you only re-embed what changed, upsert new vectors, tombstone old ones, compact periodically. Re-embedding everything nightly is a money fire.

24/35 At scale, RAG ingestion is a streaming system: source connectors, a queue, parse/chunk workers, a batched embedding service, index writers. Every arrow needs retries, idempotency, and a dead-letter queue. This is where RAG becomes engineering.

23/35 Evals are infrastructure, not a phase. Golden question set, run the pipeline, score the triad plus recall@k, gate deploys in CI. Change your chunker or embedding model without this and you're deploying vibes to production.

22/35 The RAG eval triad. Context relevance: did we fetch the right stuff? Faithfulness: does the answer stick to it? Answer relevance: did we actually address the question? A system can fail any one independently, so measure all three.

21/35 Corrective RAG: grade what you retrieved before you generate. Relevant: answer. Ambiguous: rewrite the query and retry. Irrelevant: fall back to web search or a different index, or say "I don't know". Never generate from garbage context.

20/35 Agentic RAG: put a router LLM in front of many sources: vector store, SQL, web, internal APIs. The model decides whether to retrieve, where from, and how many times, looping until it's confident. RAG stops being a pipeline and becomes a loop.

19/35 GraphRAG: extract entities and relations into a knowledge graph, cluster it into communities, pre-summarize each one. Now "what are the main themes across this corpus?" is answerable. Top-k chunk retrieval structurally cannot see that.