lionbox search

The agency stores ~15 years of client work in an enterprise Box instance. When client counsel sends a discovery request — all video and print depicting feature X, final consumer-facing only, model years A through B — one person manually hunts job folders. Slow, manual, high-stakes: legal pulls must be thorough and defensible. Box's own search string-matches filenames; it can't reason over the folder conventions (fiscal-year folders → job folders → assets, production files, traffic; a proofs folder means final print, a FINAL folder means final video) that an experienced human uses.

Parent-company IT's answer to the access request was to enable the vendor's AI agent inside the corporate assistant for a test group, while separately provisioning an API sandbox. The open decision: is the assistant sufficient for legal pulls, or do we need a custom index? The hypothesis, stated up front: conversational search will do keyword lookup well and struggle with multi-constraint structural queries, exhaustiveness, and compiling a deliverable.

One afternoon of testing settled it — fifteen logged turns. The assistant's chat API returned 403 for seven admin-consent scopes and, even consented, grounds on the office suite, not Box. The vendor's AI agent reached Box and navigated well — given a folder it listed the full job-year tree and reasoned the feature hierarchy correctly — but couldn't resolve “finals only” (Box doesn't label finals consistently), couldn't reliably produce shareable links, and when pushed to be exhaustive said it could only browse one folder at a time, offered a ~90% sample, declared it “cannot honestly certify this pull as fully exhaustive,” and errored out after twenty minutes. The failures were architectural, not tunable.

Three reframes followed. The assistant surface is the wrong layer to be the indexer — its output is prose, its links come and go, it times out; an authoritative, defensible index cannot depend on a non-deterministic intermediary. Build the index on the Box API and let the assistant be a front door. Completeness is a property of the crawl, not a judgment the agent makes — so every response carries completeness counts (3,075 considered, 162 matched, 2,913 excluded: media mismatch 1,250, out of date range 824, not final 376, …) and the operating contract forbids the LLM from certifying it. And iteration must be free: Box bills per API call, so an agent loop pointed at Box multiplies the bill by every iteration. Crawl once into a local index; then searching, faceting, and exporting make zero Box calls — enforced by test. An agent can run two hundred queries for $0.00.

Acceptance criteria before code. Written the day before the build; one pytest per criterion, named for it; status tracked as verified or claimed, with the live-Box criteria honestly left “claimed” until a real credential exists. Every design choice maps back to a Phase-1 failure ID. Seventy-four tests, including one that scans the codebase and fails the build if any Box call outside a three-method read-only allow-list appears (two crawl reads and a preflight “who am I” so the audit trail is attributable).

Cost is metered three ways. A per-call ledger by operation, persisted against the run and flushed as the crawl proceeds (a crash never loses the record). A hard ceiling checked before each call — a live crawl with no budget gets a default $25 and says so; retries are billed calls and count; the nightly job always carries the cap, because an unattended crawl on a schedule is exactly how a storage bill runs away. And a stratified pre-flight estimator that probes a few folders per depth: a 54-call probe ($0.11) projected 884 calls against 922 actual — 4% under, inside its ±20% band. The cost driver is folders, not files (one call per page of a folder's contents); the synthetic corpus — 3,075 files, 918 folders — crawls for $1.92.

Deterministic where it counts. Finals is three-state (final / likely / not final), never boolean, because inconsistent labeling is exactly what defeated the assistant; every classification carries the rule that decided it. The feature ladder (systems → versions → features with aliases and parents) is human-editable data; agents may propose terms with evidence but cannot apply them — an unreviewed expansion is an unauditable one. The ranker is additive with inspectable weights and a per-file rationale; one concept scores once, so five aliases of a feature don't outrank the feature. The crawl is resumable from a persisted frontier and an interrupted run produces a digest-identical index. Links are never fabricated; the ZIP bundle carries links, paths, classification, and method — not the files — so Box stays the access-control boundary.

Two things found only by driving the loop. A request for material the corpus didn't hold returned 107 confident-looking matches — all parent-level ladder expansions, visible only in 107 per-file rationales; an agent scanning the top results would reasonably have called them responsive. Fixed with an aggregate match-quality block (exact feature / ladder expansion / keyword only) and a loud warning when exact is zero; it became the bake-off's headline scenario. And the provider bake-off itself — four scenarios × three repeats × two providers, scored deterministically against index-derived ground truth, no LLM grading another LLM — found both models 12/12 correct including the trap, zero contract violations in 24 runs, and a provenance-disclosure gap (1.00 vs 0.17) that a prompt fix closed, plus an over-citation habit that it didn't: told explicitly not to cite unverifiable files, one model still named 7, then 15, then 30 on identical requests. It prefers producing an answer over reporting a limit — the wrong bias for discovery work. A bigger reasoning model was 36× slower for no accuracy gain; the hard part isn't reasoning depth, it's reading signals the broker already computes.

Box API → budgeted crawler (ledger, ceiling, estimator, retry-aware backoff, request pacer) → SQLite structural index (names, paths, dates, IDs — structure only in v1; content and embeddings deferred) → request parser with the feature ladder → deterministic ranker → three surfaces: a CLI, an HTTP API with eight JSON-Schema tools and the operating contract attached, and a single-page UI in the agency design system with a download-all bundle. The headless agent loop drives the same tools through either provider behind one interface, every step traced and replayable, capped at twelve. Liveness and readiness are split (503 while the index is empty); the container runs non-root with a read-only image; the nightly refresh CronJob skips cleanly when no credential is set.

The “assistant or custom index?” decision was made with evidence in one afternoon, including the assistant's own self-disqualification; the findings report to parent-company IT turned “budgeting is a big unknown” into a cost model (a projected $39–$154 for a full production index, with refresh cadence the only ongoing cost) plus built-in controls and a ten-cent probe-first plan. The broker was built and deployed the day after the criteria were written, and later wired into the fleet's telemetry with per-call dollar metering.

Measured on the synthetic corpus: 100% of in-scope files indexed with no sampling, 462 exclusions each with a rule and reason, an interrupted crawl digest-identical to an uninterrupted one, the probe within 4%, zero contract violations across the agent runs. Not yet: a real pull against the production instance — the live path is validated against a sandbox for about half a cent, but the production credential, a shared volume for the nightly job, and one root folder are pending. No time-saved figure is claimed; the human baseline is “hours, but complete and defensible,” and the design goal was to keep the second half of that sentence.