CraggleRock + RFI accelerator
RFI assistance as a real system: retrieval, chat, gateway, and a document pipeline — seven workloads, one product.
team one internal tool, deployed in production on the agency's internal aws/eks platform. sanitized to architecture level.
New-business RFIs demand accurate, well-sourced answers pulled fast from a large internal knowledge base. The default — a person hunting documents under deadline — is slow and uneven. And the obvious fix, “ask a chatbot,” hallucinates exactly when accuracy matters most.
The problem isn't adding a chatbot; it's retrieval quality and self-checking. A single-shot RAG prompt will confidently cite the adjacent-but-wrong thing. The system has to retrieve, grade its own retrieval, correct it, and only then answer — corrective RAG, not naive RAG.
The shortcut route — embed the docs, stuff top-k into a prompt, ship — fails silently on adjacent-but-wrong context. Three choices made the difference:
- Separable workloads. Document conversion, chunking, embedding, vector store, chat, gateway, client — each its own service, so each scales and fails independently, and any one can be swapped without touching the rest.
- A graded retrieval pipeline. Retrieval is scored before generation: weak results trigger a corrective fallback, weak chunks get filtered on a second threshold, and only what survives reaches the model. (Grading runs on similarity scores today — the seam for model-graded relevance is designed in.)
- An eval harness as a first-class service. Retrieval quality is measured — MRR, nDCG@k, recall@k, precision@k — and chunking strategies and thresholds are compared head-to-head with a recommended winner. RAG you don't measure is RAG you're guessing about.
Seven workloads, two public: a React client and a FastAPI gateway in front; document conversion (PDF/DOCX/PPTX/XLSX with OCR), chunking (token, propositional, or semantic strategies), embedding (768-dimension sentence transformers), the pgvector-backed store, and the chat service behind. The RFI accelerator is deliberately not a separate product — it's client routes riding the same gateway, which is the architectural point.
Source-grounded RFI responses with a self-checking retrieval layer instead of single-shot guesses — and an eval service that can prove which pipeline configuration retrieves best, instead of arguing about it. The structured critique experiences (response scoring, guardrails) ride the same retrieval spine.
LLM products are systems, not prompts. Decompose them into workloads — retrieval, chat, gateway, ingestion — and each part becomes testable, scalable, and replaceable.
the insight