Session Description

When we launched our product discovery assistant, we had no systematic way to tell when the assistant was failing a shopper, or how to fix it. These failures are usually silent: customers don’t complain, they quietly leave. At adoption scale, that’s real lost revenue.

This talk shows how we turned every production conversation into a learning signal, and wired it into a loop that runs itself every week over tens of thousands of production sessions. We walk through five building blocks, grounded in our real pipeline:

COLLECT — we export production sessions (dialog + backend funnel events) and reconstruct one timeline per session. The key trick: we merge in customer-visible funnel signals (search dispatched, product list shown, recommendation shown, guardrail fired) so the judge reasons over what the system actually did, instead of guessing from the transcript whether a search succeeded.
JUDGE — an LLM-as-judge labels each session on customer sentiment and task outcome, anchored by those signals, with channel-specific rules because text and voice can fail differently (e.g. speech-recognition errors that quietly derail a voice search).
CLUSTER — group failing sessions into single-root-cause buckets, each mapped to one product-engineering work item.
REVIEW — before we act, we confirm each cluster against the actual turn-by-turn dialogs, run a stability check and replay the session in our debugger to separate real failures from LLM artefacts. This is how we trust an LLM judge enough to act on its labels.
ACT — findings become concrete changes to retrieval and ranking tools, prompts, conversation flow, or guardrails – verified by replay against previously failing sessions. The next weekly run confirms the fix when the cluster’s share drops.

For a search audience, the failures the loop surfaces are the interesting part: over-constrained queries that return zero results or catalog and product attribute gaps. We show what we changed: from a constraint-relaxation fallback over product attribute handling. Secondarily, the same loop caught an over-sensitive guardrail cluster whose fix cut false voice blocks by double-digit percentage points. We share how these incremental fixes accumulate to move our outcome metrics (task success, sentiment, engagement, …) over time.

We’re equally honest about what was hard: LLM-judge reliability, clustering noise, and keeping humans in the loop without drowning them.

We close by mapping each building block onto standard cloud-provider services and open-source tools such as Langfuse, so you can rebuild the loop on whatever stack you already run.

Attendees leave with a concrete, reusable blueprint for an evaluation-and-improvement loop for any conversational system, including the pitfalls to avoid. The techniques (grounding an LLM judge in real signals, failure-mode clustering, stability validation, regression via scheduled re-runs) transfer whether you work on chat, voice, or agentic search.