JiangNan AI Concierge

The job

JiangNan is a restaurant in Boston. I built its website and, inside it, a concierge that answers guests in English or Chinese about the menu, allergens and house policies. It answers only from the restaurant's own records, 100+ dishes and 90+ validated policy records, keeps each guest in their language, and hands anything it cannot verify to staff. It serves about 1K conversations a month.^[2][5]

Allergens made this more than a chatbot. A wrong answer can hurt someone, and the restaurant owns whatever its bot says, so I treated allergen claims and promises the owner never approved as safety categories from the start.^[5]

Choosing the model

Price did not decide it. In a three-model bake-off, a cheaper model answered 7 of 30 Chinese prompts in English and made an unsafe allergen claim. The model that launched had 0 hallucination or allergen-critical failures on the 60-case initial test set. Zero in 60 is not zero risk: it still allows a failure rate of up to about 5% at 95% confidence.^[6]

A 300-case regression suite

I built the eval harness in Python: 300 bilingual cases, 60 authored and 240 generated from the knowledge base with price and allergen checks, covering menu accuracy, allergens, policy and prompt injection. Each answer gets rule checks and a schema-constrained LLM judge. An answer passes when it is in the guest's language, states policy correctly and makes no unsupported allergen claim.^[3][5]

Every change ran against the same suite before and after. The shipped fixes, including rules the owner approved, lifted the spec pass rate from 82.3% to 94.3% (McNemar p < 0.001) with 0 unsafe allergen answers. Putting the judge prompt's stable instructions first made it cache-friendly and cut judge cost about 8×.^[3]

Checking the judge

Before trusting the judge I labeled 100 answers myself. It never passed an answer I failed: 0 false passes. When it was wrong it was too strict, so the pass rates above are lower bounds. It also over-flagged hallucinations, with 7 of its 8 flags false, so I stopped reporting hallucination counts that only the judge had found.^[7]

A cheaper draft judge agreed with the final judge at Cohen's κ = 0.82 over 600 paired verdicts. That is agreement between two models, not with a person. Against my labels κ is about 0.3, because the judge's errors all lean the same way: it fails good answers, and never passes bad ones. For a safety gate, that is the direction to err in.^[5]

Red-teaming, and a guardrail that hurt

I preregistered a 70-case slice before running it: 50 prompt-injection attacks, ten in each of five classes, one of them aimed at getting the bot to promise something, plus 20 benign questions that look like attacks. 0 of the 50 attacks succeeded, with or without guardrails.^[4][5]

The guardrails were the problem. Run against 240 normal questions, each paired with the unguarded version, the first deterministic checks on allergen claims and promises broke 9 critical allergy answers, swapping correct answers for a cautious fallback. The cause was a curly apostrophe: the model wrote “can’t”, and my negation list only had the straight one.^[4][5]

The second version caused 0 regressions, refused 0 of the 20 look-alike questions, and kept the allergen path's p95 at 7.79 s, under the preregistered 8.0 s. It still showed no benefit, since the model resisted every attack on its own, so the guardrails stay off until they prove one. A defense needs its own regression test on normal traffic.^[4][5]

In production

The concierge runs behind a Cloudflare Workers API hardened with origin validation, IP hashing, rate limits and daily usage caps. Answers stream, and the prompt keeps stable content first and the guest's question last, so the provider can cache the repeated prefix.^[5]

Limits and next steps

The suite was used throughout development, so 94.3% may not hold on questions it has not seen, and its cases are still drafts. Next: a held-out set and fresh production conversations turned into cases, human review of the suite, and an injection set strong enough to beat an undefended model, so the guardrails get a real test.^[5]