CraggleRock + RFI accelerator

New-business RFIs demand accurate, well-sourced answers pulled fast from a large internal knowledge base. The default — a person hunting documents under deadline — is slow and uneven. And the obvious fix, “ask a chatbot,” hallucinates exactly when accuracy matters most.

The problem isn't adding a chatbot; it's retrieval quality and self-checking. A single-shot RAG prompt will confidently cite the adjacent-but-wrong thing. The system has to retrieve, grade its own retrieval, correct it, and only then answer — corrective RAG, not naive RAG.

The shortcut route — embed the docs, stuff top-k into a prompt, ship — fails silently on adjacent-but-wrong context. Three choices made the difference:

• Separable workloads. Document conversion, chunking, embedding, vector store, chat, gateway, client — each its own service, so each scales and fails independently, and any one can be swapped without touching the rest.
• A graded retrieval pipeline. Retrieval is scored before generation: weak results trigger a corrective fallback, weak chunks get filtered on a second threshold, and only what survives reaches the model. (Grading runs on similarity scores today — the seam for model-graded relevance is designed in.)
• An eval harness as a first-class service. Retrieval quality is measured — MRR, nDCG@k, recall@k, precision@k — and chunking strategies and thresholds are compared head-to-head with a recommended winner. RAG you don't measure is RAG you're guessing about.

Seven workloads, two public: a React client and a FastAPI gateway in front; document conversion (PDF/DOCX/PPTX/XLSX with OCR), chunking (token, propositional, or semantic strategies), embedding (768-dimension sentence transformers), the pgvector-backed store, and the chat service behind. The RFI accelerator is deliberately not a separate product — it's client routes riding the same gateway, which is the architectural point.

Source-grounded RFI responses with a self-checking retrieval layer instead of single-shot guesses — and an eval service that can prove which pipeline configuration retrieves best, instead of arguing about it. The structured critique experiences (response scoring, guardrails) ride the same retrieval spine.

A year in, the head of AI wanted something “more usable than current” for the new-business team — and had a lean browser prototype of his own whose “deterministic logic” he preferred. The implicit question was retrofit or rebuild; the implicit diagnosis was “the architecture is too strict” and “it needs more training data.” Reading both codebases reframed it from architecture to tuning plus one integrity gap.

It withholds because of gates, not retrieval: the corrective thresholds (0.7 relevance, 0.6 fact-check) discard usable mid-confidence context, the drafting prompt says “ground only in the provided chunks” with a low-confidence switch, and the base token budget is 512. Retrieval itself was better than its config file claimed — hybrid semantic plus BM25 fused by reciprocal rank, on 768-d embeddings, not the weaker model the config named (drift, not weakness). “Training data” meant corpus; there is no training loop. And the one real gap: citation verification in production was fake — the JSON schema asked the model to emit "verified": true and nothing checked it. The prototype verified citations in about fifteen lines of code: does the snippet exist in the source text, at what offsets, on which lines. Meanwhile a deterministic claims-review engine already existed in the platform repo — supported / unsupported / needs-disclaimer — wired to a guardrails UI but not to the drafter. Porting the prototype wholesale would have created two parallel deterministic systems.

Decision: retrofit and tune, don't rebuild. Keep retrieval, ingestion, the vector store, the topology, the deploy. Layer on, in this order: code-verified citations with a lightweight answer receipt (retrieval status, citations, missing data) landing with the threshold tuning — not after, so “too cautious” isn't traded for “confidently wrong”; helpful-by-default prompts and a consistent token budget; agency brand voice and a single per-document rigidity rule; real questionnaire auto-fill (the prototype's was a timer, not a feature); narrative and deck generation; and every change gated on the 62-question labeled eval set already sitting in the repo, so “more usable” is a number. The heavy enterprise access-control scope in a later draft was deferred until a cross-user confidential corpus actually exists. One open engineering decision was assigned rather than hand-waved: which verifier becomes the verifier — one deterministic spine, not two.

Status, honestly: diagnosed, decided, and specified to code-area level with owners; the front-end app was also recovered from a two-month “failed” deploy state (a workload slug that churned three times in one day left stale registrations — a clean redeploy cleared it). The implementation itself is pending as of this writing; the production thresholds are still 0.7 and 0.6. This page will say so until they aren't.