Building a Conversation Engine with a Narrative for Kids
A hybrid scripted and LLM conversation engine that let children aged 6–12 talk about anything while the story still moved forward. Voice-first, designed for child safety.
- Role
- Project Lead · Sole Designer · AI Architect
- Timeline
- Apr–Sep 2024
- Team
- 5 core + outsourced art
- Release
- A/B tested · Shipped
Freedom to say anything, in a story that still has to go somewhere.
Conversations are disappearing from childhood, replaced by passive screen time rather than genuine back-and-forth exchange. Research consistently links meaningful dialogue to language development and emotional intelligence in children aged 6–12. We wanted to bring those conversations back, through an AI companion children would actually talk to.
The moment we tried to build it, a fundamental tension appeared. A fully scripted character feels dead. It can't respond to what a child actually says. A fully open LLM has no sense of where the story needs to go. It wanders, forgets, and breaks character.
That tension (scripted narrative vs. generative conversation) is not unique to this project. It's the central unsolved problem in AI-driven interactive experiences. What was unique was having to solve it for children aged 6–12, using voice as the only input, with STT models that weren't built for young voices, in real-time, with a character whose integrity could not break. No existing architecture gave us what we needed.
Failure mode A / Fully scripted
Dux
“Do you have any pets at home?”
Child
“I have a robot! His name is Zorp.”
Dux ✕
“Ok now I need to tell you about the Dreamer's Lodge.”
Can't respond to what the child actually said. Dead end.
Failure mode B / Fully open LLM
Dux
“The ancient map is hidden somewhere in the Lodge!”
Child
“I love dinosaurs. Do you like T-Rexes?”
Dux ✕
“Oh yes! T-Rexes had tiny arms but were incredible hunters…”
Story gone. No concept of where the narrative needs to go.
The handoff system.
The approach we landed on is what practitioners call an FSM-style dialogue manager with bounded generative exchanges: a finite-state machine that owns the narrative, with the LLM operating inside it rather than on top of it. The state machine decides when the story moves forward. The LLM handles what happens in between.
The child could say anything inside a pocket. The narrative always moved forward when the pocket closed.
Hardcoded
01
Entry line. Narrative beat. Context injected.
LLM open
02
Two to four turns. Anything goes. The child leads.
LLM primed
03
Next beat revealed. Wind-down cued. The model closes naturally.
Hardcoded
04
Narrative resumes. Seamless transition. The story moves forward.
LLM open
05
Next pocket. New context. New topic.
LLM primed
06
And the loop closes again, in character.
- Hardcoded: the state machine owns this
- LLM primed: told what's next, winds down
- LLM open: the child leads freely
One conversation pocket, in order. The loop repeats for every beat in the story.
How one pocket closes
The same pocket closing: an illustrative exchange
01 Exit approaches
The state machine knows the pocket is about to close.
The model is never told. It does not know it has a turn limit, so its replies stay short because that is Dux's character, not because it is being throttled.
Child
“…and my robot Zorp has a laser arm!”
Dux
“A laser arm! What does he use it for?”
02 Context injected
Before the exit fires, the system sends the model an update: what the next narrative beat is, what Dux has to do to reach it, and an instruction to bring this exchange to a natural close.
Not “stop now”. “Here’s where the story is going. Guide the child there.”
Injected · never spoken
Next beat: the ancient map is hidden in the Lodge. Bring the current exchange to a close, in character.
03 The model closes itself
Dux writes his own wind-down line: noticing a sound elsewhere in the Lodge, remembering something, asking one last question.
In character, so the seam never shows. The system asked for a close; the character delivered one.
Dux · model-written
“Zorp sounds brilliant. Hey — did you hear that? Something just rattled down the hall.”
Child
“I heard it too!”
04 The hardcoded line lands
The scripted narrative beat arrives on the back of a conversation that had already started moving in that direction.
Nothing is interrupted, because the model had already begun saying goodbye.
Dux · scripted
“I think the old map might be through there. Come on — let’s go and look.”
Why we couldn't just prompt our way out of this. Every simpler option was tried first, and each failed in its own direction.
Fully scripted
Couldn't respond to what a child actually said. Every unexpected answer caused a dead end. Children aged 6–12 say unexpected things constantly.
Fully open LLM
No concept of narrative time or story structure. Dux would go so deep into a side topic that the story ceased to exist. Also: no memory, lore drift, character breaks.
Better prompting
Prompted instructions erode across a conversation. The longer the exchange, the less reliably the model followed them. A 6-year-old will always outlast a system prompt.
Existing tools (2024)
Dialogue managers like Rasa or Botpress were built for task completion, not narrative. No children's voice product had shipped a hybrid FSM + LLM architecture at this point.
The full architecture was documented and iterated on in detail before a single line was written. Every handoff point, every state transition, every fallback was mapped, because with children, edge cases are the product.
Four places the seam needed design work.
A hybrid architecture doesn't eliminate problems. It moves them. The hard part isn't the state machine or the LLM individually. It's what happens at the boundary between them. These are the four places where that boundary required the most deliberate design.
The entry line had to do two jobs
Each pocket opened with one hardcoded line that closed the previous beat for the child and primed the model's context: telling it where in the story it was, what it knew about the child, and what was in-bounds.
The LLM had to not know it was bounded
We never told the model it had a turn limit. Dux's personality (curious, one question at a time, 40 words max) made short replies feel like character, not constraint, while the exit fired quietly on turn count.
STT failure breaks the handoff
Children's voices hit word error rates 2–5× worse than adults'. Every pocket handled five input states (clear, unknown, wrong, silent, noisy) each with an in-character Dux recovery that never broke the seam.
The exit had to feel like Dux, not the system
The hardcoded exit is the most fragile moment. Every one was written as something Dux would genuinely say (a sound elsewhere in the Lodge, a callback to something the child mentioned) so the rails never showed.
The handoff only works if the character never breaks.
A hybrid architecture is invisible to the user if (and only if) the character stays consistent across the seam. Every transition between scripted and open had to feel like Dux, not like a system state change. That required every element of Dux's design to reinforce the same coherent personality.
Three states. One character. Every voice interaction had to signal what Dux was doing. Without a clear visual state, children didn't know when to talk. Getting the latency moment right (turning processing time into visible character behaviour) was one of the smallest but most impactful design decisions in the whole project.
Solving the blank page. Open pockets only work if the child has something to say. In testing, children went quiet not because they were shy, but because “you can talk about anything” is a harder prompt than it sounds. Topic Cards gave the child a visible starting point: something to choose, not something to be told. The pocket still opened. The conversation still flowed. We just removed the blank page.
The proof that the seam was invisible.
Children started talking to Dux before we taught them how.
No instruction. No onboarding prompt. They simply started talking, because Dux felt like someone, not a feature. That signal tells you the handoff architecture was working: if children had noticed the transition between scripted and open conversation, the illusion would have broken. They didn't notice.
A second observation confirmed it further: when the Dreamer's Lodge opened up and children had an entire world to explore, they kept coming back to Dux. The character held attention even competing against the environment. The architecture kept the story coherent. The character made children want to stay in it.
You don't design the model. You design around it.
Designing AI products is not about designing the model. It's about designing the systems around the model.
The LLM was one component. The state machine, the context injectors, the hardcoded entry and exit lines, the character poses that mapped to pipeline states, the in-character STT fallbacks, the topic cards that solved the blank page. None of that is in the model. All of it is design. And all of it had to work together for the handoff to be invisible.
The hybrid scripted/LLM architecture is now a known pattern across EdTech, game AI, and voice products. What isn't common is designing that architecture for children: where STT fails constantly, where narrative coherence has to survive a six-year-old, and where the character's integrity is the product. That specificity is what made it hard, and worth solving.