Jeremy Yao

Grounded in the house rules

The concierge answers from the restaurant's own menu and service policies: 100+ dishes and 90+ validated policy records.

Language routing and allergen safeguards

Guests write in English or Chinese and get answers in their language, with allergen safeguards and escalation to staff. It served 1K conversations a month.

A three-model bake-off

A cheaper model answered 7 of 30 Chinese prompts in English and made an unsafe allergen claim, so it lost. The chosen model had 0 of 60 hallucination or allergen-critical failures on the initial test set.

300 bilingual eval cases

60 authored cases and 240 generated from the knowledge base with price and allergen checks. Owner-approved rules lifted spec pass rate from 82.3% to 94.3% (McNemar p < 0.001), with 0 unsafe allergen answers.

Auditing the judge

Cache-stable prefixes cut judge cost to about 1/8. Against 100 human labels the judge never passed a bad answer but over-flagged (1 of 8 hallucination flags held up), so pass rates count as conservative. A draft judge matched the final at κ = 0.82 over 600 verdicts.

Red team, and guards that had to earn their place

0 of 50 prompt-injection attacks succeeded in a preregistered 70-case slice. The v1 guards caused 9 critical allergy regressions on 240 paired controls; v2 fixed them, and guards stayed off until proven useful.

Request routed

Model routing sends each request to the first model in an ordered fallback list.

Streaming, separated

Text, reasoning, and attachment events travel as separate SSE event types.

Failure classified

Rate limits, 5xx responses, timeouts, and connection errors are retryable; an invalid 4xx fails fast.

Bounded retry, traced

Retries are bounded, and every attempt is traced with structured logs and OpenTelemetry counters.

Ordered fallback

When retries run out, the request moves to the next model in order.

Recovered after reconnect

After a reconnect, live client drafts are reconciled with persisted PostgreSQL messages to restore state, without passing stored reasoning into subsequent model context.