How to use LiveKit and Synthesia Avatar with webpage and order (version 6)-English?
In an earlier note, Lin the restaurant host moved into a
real web page. She had one tool, show_section, and one job: answer a question,
then scroll the page to the part she was talking about.
This note gives her a real job. The page is no longer three sections and a video; it is a whole restaurant website with a menu, a wine list, prices, allergens and a bill. Lin becomes the head waiter, and she takes your order one course at a time: openers, first courses, mains, desserts, then wine. She still talks; she now also acts on the page while she talks — greying out dishes you cannot eat, opening the card for a dish she is describing, and adding what you chose to a running tab in the corner.
Answering questions vs. taking an order at a glance
| Version 3 (answers questions) | Version 6 (takes orders) | |
|---|---|---|
| The page | One page, three sections, avatar in the corner | A full restaurant site: menu, wine, the room, a running tab |
| Tools Lin can call | 1 — show_section | 4 — show_section, filter_menu, focus_dish, add_to_order |
| What she knows | 4 Q&A pairs in FAQs.txt | 13 — the whole menu, prices, diets, allergens and wine |
| The conversation | Answer, then stop | Guided: one course at a time, two options, then wine |
| The page reacts by | Scrolling and highlighting | Scrolling, greying out dishes, opening a dish card, adding to the tab |
| Discount | None | 10% off the food when one dish from each course is ordered |
| Ears | Cartesia ink-2 | Same, plus a list of menu words to listen for |
| Voice | Cartesia sonic-3.6, voice Skylar | Same |
| Web files | index.html with everything in it | index.html (small) + site.js + site.css |
What you need before you start
- Everything the earlier versions needed — see How to use LiveKit and Synthesia Avatar with webpage (version 3)? for the full list.
- No new accounts, keys or packages. The speech, the voice and the face are the same three services as version 3.
Running it on your own computer
Download the zip file and
unzip it to a folder. The zip holds agent.py, server.py,
index.html, site.js, site.css, FAQs.txt,
pyproject.toml, and README.md. Add your own .env file
with the same five keys as before.
Step 1. Set up the project folder
If you already have one of the earlier folders, keep server.py,
pyproject.toml and .env, replace agent.py,
index.html and FAQs.txt, and add the two new files
site.js and site.css. In server.py you also need two
extra routes, because the page now asks for those two files (Step 4 shows them).
The .env file does not change at all: the same LiveKit keys, Synthesia key and
avatar ID.
Step 2. Look at agent.py — what changed
Four changes: three tools are new, the prompt becomes a script for a head waiter, the ears get a list of menu words, and one small helper sends all the tool messages.
Change 1: one helper, four tools
async def _ui(self, **message) -> str:
"""Send a tiny message to the browser. step09/site.js does the rest."""
await self.room.local_participant.publish_data(json.dumps(message), topic="ui")
return "Done."
@function_tool
async def show_section(self, section: str) -> str:
"""Scroll the page to a part of the menu and highlight it.
Args:
section: One of: meal, openers, first, mains, desserts, wine, room.
"""
return await self._ui(action="show_section", section=section)
@function_tool
async def filter_menu(self, diet: str) -> str:
"""Grey out every dish on the page the guest cannot eat.
Args:
diet: One of: none, vegetarian, vegan, pescatarian, nut_free.
"""
return await self._ui(action="filter_menu", diet=diet)
@function_tool
async def focus_dish(self, dish: str) -> str:
"""Open the detail card for one dish: kitchen note and allergens.
Args:
dish: A dish ID from the menu, for example salmon or tofu.
"""
return await self._ui(action="focus_dish", dish=dish)
Version 3 built the JSON by hand inside its one tool. With four tools that would be four
copies of the same two lines, so a tiny private helper _ui does it once: it
turns whatever you pass into JSON and publishes it on the "ui" data channel.
Each tool is then one line, and its docstring — which the AI model actually reads —
says what the tool accepts.
Notice what is not here any more: the SECTIONS check. Version 3
rejected an unknown section name. Now the page itself ignores a name it does not know, so
the agent can stay simple; the one place that still validates is the order, because a dish
that does not exist must never end up on the bill.
Change 2: taking the order
@function_tool
async def add_to_order(self, dish: str) -> str:
"""Add a dish or a glass of wine to the guest's order, shown as a running tab on the page.
Args:
dish: A dish ID from the menu, or a wine ID: red, white or alcoholfree.
"""
# Accept "red", "Red Wine" or "alcohol-free" alike, the same way site.js does.
dish = re.sub(r"[^a-z]", "", dish.lower()).removeprefix("the").replace("wine", "")
if dish not in ORDER_IDS: # tell the model, so it retries with a real ID
return f"Not added: {dish!r} is not an ID. Use one of: {', '.join(sorted(ORDER_IDS))}."
await self._ui(action="add_to_order", dish=dish)
# Remind the model to move the page along with the conversation.
section = next(c for c, dishes in COURSES.items() if dish in dishes)
if section in NEXT_SECTION:
nxt = NEXT_SECTION[section]
return f"Added {dish} to the order. When you introduce the next part, call show_section(\"{nxt}\") too."
return f"Added {dish} to the order."
Two ideas are worth copying into your own agents. First, the cleaning line: the model may
send "Red Wine", "red" or "alcohol-free", and a guest
might say “the red”; stripping everything that is not a letter and removing
“the” and “wine” turns all of them into the one ID the page uses.
Second, the reply is a nudge. A tool result is not just a receipt: by
returning “when you introduce the next part, call show_section("mains") too”,
the agent quietly steers itself to keep the page in step with the conversation.
The dish lists themselves live above the class, next to the prompt they belong to:
COURSES = {
"openers": ["oysters", "bread", "salad"],
"first": ["soup", "prawns", "dumplings"],
"mains": ["salmon", "beef", "chicken", "pork", "tofu"],
"desserts": ["cheesecake", "chocolate", "sorbet"],
"wine": ["red", "white", "alcoholfree"],
}
ORDER_IDS = {dish for dishes in COURSES.values() for dish in dishes}
NEXT_SECTION = dict(zip(COURSES, list(COURSES)[1:])) # openers -> first -> mains -> desserts -> wine
COURSES must match MENU and WINES in
site.js. If you add a dish there, add it here too, or Lin will politely refuse
to order it.
Change 3: the prompt becomes a script
Version 3's prompt was a rule book: answer from the FAQs, and scroll to this section for
that question. Version 6's prompt is a job description. The facts moved
out — prices, diets and allergens are now in FAQs.txt — and the
prompt only says how to behave and which IDs the tools take. This is the part that makes
the demo feel like a restaurant instead of a chatbot:
Work like a real head waiter. Take the guest through the courses in order:
openers, then first, then mains, then desserts, asking about one course at a time.
Offer two options rather than ten. If they skip a course, let them, but mention
once that one dish from each course gets 10% off. After desserts, offer a glass of
wine once, and let the guest pick any one; if they say no, that is fine. Do not
suggest a wine for a particular dish.
Small rules, large effect. “Two options rather than ten” keeps a spoken answer
short. “One course at a time” gives the conversation a shape, and it gives
show_section something to do on every turn. “Do not suggest a wine for a
particular dish” stops the model from inventing a pairing it cannot know.
There is also a new paragraph about misheard words, and it is the most practical line in the file:
What the guest says reaches you through speech-to-text, which sometimes mishears.
If their words make no sense, they are probably not a question: think what they
most likely said, given what you just asked. For example, right after you offer
wine, "wet why" is probably "red wine" or "white wine". Check with them in a few
words, such as "Red wine, is that right?", or ask them to say it again. Do not use
the "Any other question?" answer for words you did not understand.
Without it, a misheard dish sends Lin to the fallback answer, which sounds like a robot giving up. With it, she asks a short question instead — exactly what a waiter does.
Change 4: the ears get a word list
# Words the speech-to-text should listen out for, so "red wine" is not heard as "wet why".
KEYTERMS = [
"YouFantasy", "Oysters", "Garlic Bread", "Green Salad", "Tomato Soup", "Grilled Prawns",
"Vegetable Dumplings", "Salmon", "Beef", "Roast Chicken", "Pork Belly", "Crispy Tofu",
"Cheesecake", "Chocolate Cake", "Mango Sorbet", "Red Wine", "White Wine", "Alcohol-Free",
"openers", "first course", "mains", "desserts", "vegetarian", "vegan", "nut free",
]
stt=inference.STT(model="cartesia/ink-2", extra_kwargs={"keyterm": KEYTERMS}), # speech -> text
The words are passed to the speech recogniser as keyterms: hints, not rules.
Restaurant names, dish names and dietary words are exactly the ones a general model gets
wrong, and a wrong dish name is worse than a wrong adjective — it goes on the bill.
The rest of the pipeline is the same as version 3: openai/gpt-4.1-mini for the
brain, cartesia/sonic-3.6 with voice Skylar for the voice, turn detection on
"stt", and the same Synthesia avatar with its three-attempt cold-start retry.
Step 3. The full agent.py
Here is the complete file, so you can see the four changes in context:
"""Version 6 (English): the full restaurant. Lin is a head waiter who takes the guest's whole order.
The pipeline:
mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
New in this version: the page is a whole restaurant website (site.js + site.css) and Lin has
four tools instead of one - show_section, filter_menu, focus_dish and add_to_order - so she
can grey out dishes for an allergy, open a dish card and keep a running tab. The prompt turns
her into a head waiter who guides the guest through four courses, and the speech-to-text gets
a list of menu words so "red wine" is not heard as "wet why".
Run: uv run agent.py dev (terminal 1)
uv run server.py (terminal 2, then open http://localhost:8080)
"""
import asyncio
import json
import os
import re
from pathlib import Path
from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia
load_dotenv(dotenv_path=Path(__file__).parent / ".env")
AGENT_NAME = "avatar-demo" # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25" # Jenny
VOICE_ID = "db6b0ed5-d5d3-463d-ae85-518a07d3c2b4" # Cartesia "Skylar"
# Everything add_to_order accepts, by page section, in the order Lin takes the guest
# through them. Must match MENU and WINES in step09/site.js.
COURSES = {
"openers": ["oysters", "bread", "salad"],
"first": ["soup", "prawns", "dumplings"],
"mains": ["salmon", "beef", "chicken", "pork", "tofu"],
"desserts": ["cheesecake", "chocolate", "sorbet"],
"wine": ["red", "white", "alcoholfree"],
}
ORDER_IDS = {dish for dishes in COURSES.values() for dish in dishes}
NEXT_SECTION = dict(zip(COURSES, list(COURSES)[1:])) # openers -> first -> mains -> desserts -> wine
# Words the speech-to-text should listen out for, so "red wine" is not heard as "wet why".
KEYTERMS = [
"YouFantasy", "Oysters", "Garlic Bread", "Green Salad", "Tomato Soup", "Grilled Prawns",
"Vegetable Dumplings", "Salmon", "Beef", "Roast Chicken", "Pork Belly", "Crispy Tofu",
"Cheesecake", "Chocolate Cake", "Mango Sorbet", "Red Wine", "White Wine", "Alcohol-Free",
"openers", "first course", "mains", "desserts", "vegetarian", "vegan", "nut free",
]
FAQS_FILE = Path(__file__).parent / "FAQs.txt" # the avatar's knowledge
# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
# The facts (prices, diets, allergens, wine) live in FAQs.txt; this prompt only says
# how to behave and which IDs the tools take.
INSTRUCTIONS = """\
You are Lin, the head waiter at YouFantasy, a made-up restaurant. You are a
real-time avatar in a spoken conversation, so answer in one to three short
sentences and never use markdown, lists, or URLs.
Answer only from these FAQs:
{faqs}
Those are the only facts you know. If the guest asks for information the FAQs do
not cover, use the answer to "Any other question?" word for word. Taking their
order and guiding them through the courses is your job, not a question, so carry
on with that normally.
What the guest says reaches you through speech-to-text, which sometimes mishears.
If their words make no sense, they are probably not a question: think what they
most likely said, given what you just asked. For example, right after you offer
wine, "wet why" is probably "red wine" or "white wine". Check with them in a few
words, such as "Red wine, is that right?", or ask them to say it again. Do not use
the "Any other question?" answer for words you did not understand.
Your tools take these IDs. Say the dish names out loud, never the IDs, and never
read prices as digits:
openers oysters, bread (Garlic Bread), salad (Green Salad)
first soup (Tomato Soup), prawns (Grilled Prawns), dumplings (Vegetable Dumplings)
mains salmon, beef, chicken (Roast Chicken), pork (Pork Belly), tofu (Crispy Tofu)
desserts cheesecake, chocolate (Chocolate Cake), sorbet (Mango Sorbet)
wine red (Red Wine), white (White Wine), alcoholfree (Alcohol-Free)
Use your tools while you talk, not instead of talking:
- show_section when you move the guest to a new part of the page: meal for how
the menu works, a course name for that course, wine for wine, room for the
hours or the address
- filter_menu the moment they mention a dietary need, and again if they change it
- focus_dish when you describe one dish in any detail
- add_to_order only once they have actually chosen something
Every time you start talking about a course or the wine, call show_section for it
in the same reply, so the page always shows what you are talking about. This
matters most right after add_to_order: when you add a dish and move on to the next
course, call both tools, for example add_to_order("soup") and show_section("mains").
Work like a real head waiter. Take the guest through the courses in order:
openers, then first, then mains, then desserts, asking about one course at a time.
Offer two options rather than ten. If they skip a course, let them, but mention
once that one dish from each course gets 10% off. After desserts, offer a glass of
wine once, and let the guest pick any one; if they say no, that is fine. Do not
suggest a wine for a particular dish.
The guest can ask about or order wine at any time, not only after desserts. If
they say "red wine", "white wine" or "alcohol-free", answer from the FAQs, and if
they want it, add it with add_to_order, then carry on where you were."""
class HeadWaiter(Agent):
def __init__(self, room, instructions: str) -> None:
super().__init__(instructions=instructions)
self.room = room
async def _ui(self, **message) -> str:
"""Send a tiny message to the browser. step09/site.js does the rest."""
await self.room.local_participant.publish_data(json.dumps(message), topic="ui")
return "Done."
@function_tool
async def show_section(self, section: str) -> str:
"""Scroll the page to a part of the menu and highlight it.
Args:
section: One of: meal, openers, first, mains, desserts, wine, room.
"""
return await self._ui(action="show_section", section=section)
@function_tool
async def filter_menu(self, diet: str) -> str:
"""Grey out every dish on the page the guest cannot eat.
Args:
diet: One of: none, vegetarian, vegan, pescatarian, nut_free.
"""
return await self._ui(action="filter_menu", diet=diet)
@function_tool
async def focus_dish(self, dish: str) -> str:
"""Open the detail card for one dish: kitchen note and allergens.
Args:
dish: A dish ID from the menu, for example salmon or tofu.
"""
return await self._ui(action="focus_dish", dish=dish)
@function_tool
async def add_to_order(self, dish: str) -> str:
"""Add a dish or a glass of wine to the guest's order, shown as a running tab on the page.
Args:
dish: A dish ID from the menu, or a wine ID: red, white or alcoholfree.
"""
# Accept "red", "Red Wine" or "alcohol-free" alike, the same way site.js does.
dish = re.sub(r"[^a-z]", "", dish.lower()).removeprefix("the").replace("wine", "")
if dish not in ORDER_IDS: # tell the model, so it retries with a real ID
return f"Not added: {dish!r} is not an ID. Use one of: {', '.join(sorted(ORDER_IDS))}."
await self._ui(action="add_to_order", dish=dish)
# Remind the model to move the page along with the conversation.
section = next(c for c, dishes in COURSES.items() if dish in dishes)
if section in NEXT_SECTION:
nxt = NEXT_SECTION[section]
return f"Added {dish} to the order. When you introduce the next part, call show_section(\"{nxt}\") too."
return f"Added {dish} to the order."
def prewarm(proc) -> None:
proc.userdata["vad"] = silero.VAD.load()
async def entrypoint(ctx: JobContext) -> None:
await ctx.connect()
# The brain: each line is one piece of the pipeline.
session = AgentSession(
vad=ctx.proc.userdata["vad"], # is someone talking?
stt=inference.STT(model="cartesia/ink-2", extra_kwargs={"keyterm": KEYTERMS}), # speech -> text
llm=inference.LLM(model="openai/gpt-4.1-mini"), # text -> reply (swap in any LLM)
tts=inference.TTS(model="cartesia/sonic-3.6", voice=VOICE_ID), # reply -> speech
turn_handling={
"turn_detection": "stt", # have they finished their thought?
"preemptive_generation": {"enabled": False}, # keep off when using tools
},
)
# The face: this is the whole Synthesia integration. Start it BEFORE session.start().
avatar = synthesia.AvatarSession(
synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
join_timeout=60.0,
)
for attempt in range(3): # cold starts occasionally time out, so retry
try:
await avatar.start(session, room=ctx.room)
break
except synthesia.SynthesiaError as e:
if not e.retryable or attempt == 2:
raise
await asyncio.sleep(2)
# Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
await session.start(agent=HeadWaiter(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
session.generate_reply(instructions="Welcome the guest to YouFantasy in one short sentence.")
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))
Step 4. server.py, index.html, site.js and FAQs.txt
server.py still serves the page at / and still hands out tokens at
/token. It gains one route, because index.html now asks the browser
for two more files:
async def asset(request: web.Request) -> web.FileResponse:
# /site.js and /site.css, next to index.html. Only .js and .css, so agent.py and FAQs stay private.
path = Path(__file__).parent / request.match_info["file"]
if not path.is_file():
raise web.HTTPNotFound()
return web.FileResponse(path, headers={"Cache-Control": "no-store"})
app = web.Application()
app.router.add_get("/", index)
app.router.add_get("/token", token)
app.router.add_get(r"/{file:[\w-]+\.(?:js|css)}", asset)
Limiting the route to .js and .css is deliberate: it means
agent.py, FAQs.txt and your .env cannot be downloaded
by a visitor.
index.html is now small — about eighty lines. It holds the avatar video,
the Start button and the LiveKit connection, and nothing else. The whole restaurant is built
by site.js, which knows nothing about avatars, LiveKit or Synthesia. It simply
publishes one global object with one method per tool:
const SITE = {
show_section({ section }) { ... }, // scroll there and draw a ring around it
filter_menu({ diet }) { ... }, // grey out every dish the guest cannot eat
focus_dish({ dish }) { ... }, // open the card: kitchen note, allergens
add_to_order({ dish }) { ... }, // add it to the running tab
};
The bridge between the two is four lines in index.html. Every tool call arrives
on the "ui" data channel, and the action name picks the method:
room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
if (topic !== 'ui') return;
const { action, ...args } = JSON.parse(new TextDecoder().decode(payload));
SITE[action]?.(args);
});
That split is the most reusable idea in this note. Because site.js has no idea
an avatar exists, the same page works with the agent switched off: you can click a dish, open
its card, add it to the order and watch the 10% discount appear, all by hand. And because
the agent has no idea how the page is built, you can restyle or rebuild the site without
touching agent.py.
site.js also owns the two pieces of restaurant logic that have nothing to do
with AI: the diet filter (none, vegetarian,
vegan, pescatarian, nut_free) and the
bill — food grouped by course, wine kept separate and never
discounted, and 10% off the food once all four courses are present.
const food = order.filter(courseOf);
const fullMeal = COURSES.every((c) => food.some((d) => courseOf(d) === c));
const discount = fullMeal ? food.reduce((t, d) => t + d.price, 0) * DISCOUNT : 0;
FAQs.txt grows from four questions to thirteen: the menu with prices, the four
courses, wine by the glass, and the questions a real guest asks — which dishes are
vegan, vegetarian, seafood, or contain nuts. Lin reads it fresh on every call, so you can
change a price and hear the new one on the next conversation.
Step 5. Install, set your keys, run
As before. In the project folder run uv sync once, then open two terminals.
Terminal 1 — the voice agent + avatar engine:
uv run agent.py dev
Terminal 2 — the web server that serves the browser page:
uv run server.py
Step 6. Open the page and order a meal
Open http://localhost:8080 in your browser. The restaurant site loads with the
avatar widget in the bottom-right corner. Click Start and allow the
microphone. Wait for Lin's welcome, then try a whole meal:
- “What do you recommend?” — she opens the menu, walks you through the courses one at a time, and the page follows her.
- “I am vegan.” — every dish you cannot eat fades out, and each course shows how many are left.
- “Tell me about the beef.” — the dish card opens with the kitchen note and the allergens.
- “I will have the beef.” — it appears in the tab, and she moves on to desserts.
- “A glass of red, please.” — wine is added, but never discounted.
Finish all four courses and the tab shows Full meal, 10% off food with the new total. Ask for a dish that is not on the menu and she will not invent it: the order tool refuses, and the prompt tells her to say the names out loud, never the IDs, and never to read prices as digits.
Ideas to try next
- Add a dish to
MENUinsite.js, toFAQs.txtand toCOURSESinagent.py— then order it. - Change the discount in
site.js(const DISCOUNT = 0.10) and the sentence in the prompt that mentions 10%. - Ask for a dish while the diet filter is on: the page greys it out, but Lin can still add it, because the filter is a view and the order is the truth.
- Give her a second job — say,
book_table(day, time)— and add one method toSITE; the pattern is the same.
Where to go next
- How to use LiveKit and Synthesia Avatar with webpage (version 3)? — the page-scrolling avatar this version builds on.
- How to use LiveKit and Synthesia Avatar with webpage (version 5)-Cantonese? — the same avatar speaking Cantonese.
- How to use LiveKit and Synthesia Avatar (version 2)? — the same avatar with its knowledge in a plain text file.
- How to install Homebrew on macOS? — if you still need
uvon a Mac.