ContextRAG

The Problem

LLMs hallucinate when answering over private documents. Teams need answers traceable to source evidence — not plausible guesses — across messy real-world formats like PDFs, spreadsheets, and slide decks.

The Approach

Two pipelines. An ingestion pipeline turns 15 document formats into parent-child chunk structures with 2048-dim embeddings. A multi-stage query pipeline fuses dense and sparse retrieval, reranks with a cross-encoder, and generates strict-JSON answers whose numbered citations are validated end-to-end by the prompt, the normalizer, and the renderer.

Architecture

1. 01

   File-first ingestion

   One unified pipeline ingests 15 formats (PDF, DOCX, XLSX, PPTX, MD, TXT, JSON, CSV, images) via Xberg extraction, with a live 4-stage pipeline visualizer in the UI.

2. 02

   Structural chunking

   A custom AST-like chunker respects markdown headers, tables, and code fences — producing parent sections for LLM context and child chunks for precise retrieval.

3. 03

   2048-dim embeddings

   Child chunks are embedded with NVIDIA NIM (nemotron-3-embed-1b) and stored in MongoDB Atlas with auto-provisioned vector and full-text indexes.

4. 04

   Hybrid retrieval

   Dense vector search (Atlas $vectorSearch, cosine) runs in parallel with sparse BM25 keyword search (Atlas Lucene) to catch both semantic and exact-match signals.

5. 05

   Reciprocal Rank Fusion

   RRF merges dense and sparse ranked lists, then hydration attaches parent context and deduplicates — keeping the best child per parent section.

6. 06

   Cross-encoder reranking

   Cohere Rerank v4.0 Pro rescores fused candidates for fine-grained relevance and prunes weak context before generation.

7. 07

   Enforced citation contract

   Gemma 4 (31B-IT) emits strict JSON answers with numbered evidence IDs — validated by prompt, normalizer, and renderer, so every claim maps to a retrievable source chunk.

Key Highlights

• Hybrid retrieval — 2048-dim Atlas Vector Search fused with BM25 via Reciprocal Rank Fusion
• Cohere Rerank v4.0 Pro cross-encoder prunes irrelevant context before generation
• Enforced citation contract with per-answer citation-match score and an interactive evidence drawer

What I Learned

• Structured-output contracts turn LLMs from plausible guessers into verifiable systems — validation belongs at every layer, not just the prompt.
• RRF + cross-encoder reranking is a pragmatic default: cheap recall from two channels, then expensive precision applied to a small candidate set.
• Production hardening matters as much as the pipeline — security headers, rate limiting, gzip, centralized retries with backoff, and soft deletes across the knowledge layer.