natural-language data lake search
Ask the media data lake a question in plain English; it finds the right view, writes the SQL, and answers.
team one internal tool, deployed in production on the agency's internal aws/eks platform. client data specifics sanitized.
A large media-analytics data lake held the answers, but getting them meant an analyst building a view and writing SQL. Non-technical stakeholders queued behind that bottleneck for questions that should take seconds. The recurring tax wasn't any single query — it was the analyst-per-question dependency.
The ask sounded like “build more dashboards.” The real opportunity was removing the translation layer entirely: let the question be the interface. And the hard part isn't generating SQL — it's grounding the query in what actually exists, so the answer is trustworthy instead of plausible.
The naive route — point an LLM at the warehouse and let it write SQL — produces confident wrong answers and a security surface nobody wants. Two choices made this one different:
- The catalog is the ground truth. Postgres stores a map of what's available — every reporting view, its columns, its metrics, and per-dimension coverage with sample values — not a copy of the data. A refresh job probes all ~75 views and records which actually carry data, so the agent only ever reaches for views that can answer.
- Blast radius: zero. Generated SQL never touches the warehouse. The agent pulls the live view it chose, loads it into an in-memory SQLite copy, and runs its SQL there — with hard caps on steps, views, time, and rows. The worst a bad query can do is waste its own sandbox.
A ReAct-style agent loop: the model gets the rendered catalog and the question, chooses views to load, pulls live data through the BI platform's REST API, writes and executes SQL against the in-memory copy, observes results, and iterates — errors go back into the transcript as observations, so it self-corrects. The final answer ships with a coverage caveat and the full SQL trace. Provider-agnostic LLM layer; responses stream over SSE so long runs survive the load balancer's idle timeout.
Self-serve answers from the data lake in non-technical hands — the analyst-per-question dependency removed for the questions that never needed an analyst. The interface states its own limits: a “what this does & doesn't see” panel, and a coverage note on every answer. Self-serve analytics earns trust by saying what it can't see.
Map the question to the catalog, not the data. Postgres holds the map of what's available; the LLM writes SQL against structure it can verify.
the insight