English 繁體中文 简体中文

How to use LiveKit and Synthesia Avatar with webpage and order (version 6)-English?

Howard Lam, PhD · AI Notes · November 6, 2026 · 9 min read

In an earlier note, Lin the restaurant host moved into a real web page. She had one tool, show_section, and one job: answer a question, then scroll the page to the part she was talking about.

This note gives her a real job. The page is no longer three sections and a video; it is a whole restaurant website with a menu, a wine list, prices, allergens and a bill. Lin becomes the head waiter, and she takes your order one course at a time: openers, first courses, mains, desserts, then wine. She still talks; she now also acts on the page while she talks — greying out dishes you cannot eat, opening the card for a dish she is describing, and adding what you chose to a running tab in the corner.

Answering questions vs. taking an order at a glance

Version 3 (answers questions)Version 6 (takes orders)
The pageOne page, three sections, avatar in the cornerA full restaurant site: menu, wine, the room, a running tab
Tools Lin can call1 — show_section4 — show_section, filter_menu, focus_dish, add_to_order
What she knows4 Q&A pairs in FAQs.txt13 — the whole menu, prices, diets, allergens and wine
The conversationAnswer, then stopGuided: one course at a time, two options, then wine
The page reacts byScrolling and highlightingScrolling, greying out dishes, opening a dish card, adding to the tab
DiscountNone10% off the food when one dish from each course is ordered
EarsCartesia ink-2Same, plus a list of menu words to listen for
VoiceCartesia sonic-3.6, voice SkylarSame
Web filesindex.html with everything in itindex.html (small) + site.js + site.css

What you need before you start

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds agent.py, server.py, index.html, site.js, site.css, FAQs.txt, pyproject.toml, and README.md. Add your own .env file with the same five keys as before.

Step 1. Set up the project folder

If you already have one of the earlier folders, keep server.py, pyproject.toml and .env, replace agent.py, index.html and FAQs.txt, and add the two new files site.js and site.css. In server.py you also need two extra routes, because the page now asks for those two files (Step 4 shows them).

The .env file does not change at all: the same LiveKit keys, Synthesia key and avatar ID.

Step 2. Look at agent.py — what changed

Four changes: three tools are new, the prompt becomes a script for a head waiter, the ears get a list of menu words, and one small helper sends all the tool messages.

Change 1: one helper, four tools

async def _ui(self, **message) -> str:
    """Send a tiny message to the browser. step09/site.js does the rest."""
    await self.room.local_participant.publish_data(json.dumps(message), topic="ui")
    return "Done."

@function_tool
async def show_section(self, section: str) -> str:
    """Scroll the page to a part of the menu and highlight it.

    Args:
        section: One of: meal, openers, first, mains, desserts, wine, room.
    """
    return await self._ui(action="show_section", section=section)

@function_tool
async def filter_menu(self, diet: str) -> str:
    """Grey out every dish on the page the guest cannot eat.

    Args:
        diet: One of: none, vegetarian, vegan, pescatarian, nut_free.
    """
    return await self._ui(action="filter_menu", diet=diet)

@function_tool
async def focus_dish(self, dish: str) -> str:
    """Open the detail card for one dish: kitchen note and allergens.

    Args:
        dish: A dish ID from the menu, for example salmon or tofu.
    """
    return await self._ui(action="focus_dish", dish=dish)

Version 3 built the JSON by hand inside its one tool. With four tools that would be four copies of the same two lines, so a tiny private helper _ui does it once: it turns whatever you pass into JSON and publishes it on the "ui" data channel. Each tool is then one line, and its docstring — which the AI model actually reads — says what the tool accepts.

Notice what is not here any more: the SECTIONS check. Version 3 rejected an unknown section name. Now the page itself ignores a name it does not know, so the agent can stay simple; the one place that still validates is the order, because a dish that does not exist must never end up on the bill.

Change 2: taking the order

@function_tool
async def add_to_order(self, dish: str) -> str:
    """Add a dish or a glass of wine to the guest's order, shown as a running tab on the page.

    Args:
        dish: A dish ID from the menu, or a wine ID: red, white or alcoholfree.
    """
    # Accept "red", "Red Wine" or "alcohol-free" alike, the same way site.js does.
    dish = re.sub(r"[^a-z]", "", dish.lower()).removeprefix("the").replace("wine", "")
    if dish not in ORDER_IDS:  # tell the model, so it retries with a real ID
        return f"Not added: {dish!r} is not an ID. Use one of: {', '.join(sorted(ORDER_IDS))}."
    await self._ui(action="add_to_order", dish=dish)
    # Remind the model to move the page along with the conversation.
    section = next(c for c, dishes in COURSES.items() if dish in dishes)
    if section in NEXT_SECTION:
        nxt = NEXT_SECTION[section]
        return f"Added {dish} to the order. When you introduce the next part, call show_section(\"{nxt}\") too."
    return f"Added {dish} to the order."

Two ideas are worth copying into your own agents. First, the cleaning line: the model may send "Red Wine", "red" or "alcohol-free", and a guest might say “the red”; stripping everything that is not a letter and removing “the” and “wine” turns all of them into the one ID the page uses. Second, the reply is a nudge. A tool result is not just a receipt: by returning “when you introduce the next part, call show_section("mains") too”, the agent quietly steers itself to keep the page in step with the conversation.

The dish lists themselves live above the class, next to the prompt they belong to:

COURSES = {
    "openers": ["oysters", "bread", "salad"],
    "first": ["soup", "prawns", "dumplings"],
    "mains": ["salmon", "beef", "chicken", "pork", "tofu"],
    "desserts": ["cheesecake", "chocolate", "sorbet"],
    "wine": ["red", "white", "alcoholfree"],
}
ORDER_IDS = {dish for dishes in COURSES.values() for dish in dishes}
NEXT_SECTION = dict(zip(COURSES, list(COURSES)[1:]))  # openers -> first -> mains -> desserts -> wine

COURSES must match MENU and WINES in site.js. If you add a dish there, add it here too, or Lin will politely refuse to order it.

Change 3: the prompt becomes a script

Version 3's prompt was a rule book: answer from the FAQs, and scroll to this section for that question. Version 6's prompt is a job description. The facts moved out — prices, diets and allergens are now in FAQs.txt — and the prompt only says how to behave and which IDs the tools take. This is the part that makes the demo feel like a restaurant instead of a chatbot:

Work like a real head waiter. Take the guest through the courses in order:
openers, then first, then mains, then desserts, asking about one course at a time.
Offer two options rather than ten. If they skip a course, let them, but mention
once that one dish from each course gets 10% off. After desserts, offer a glass of
wine once, and let the guest pick any one; if they say no, that is fine. Do not
suggest a wine for a particular dish.

Small rules, large effect. “Two options rather than ten” keeps a spoken answer short. “One course at a time” gives the conversation a shape, and it gives show_section something to do on every turn. “Do not suggest a wine for a particular dish” stops the model from inventing a pairing it cannot know.

There is also a new paragraph about misheard words, and it is the most practical line in the file:

What the guest says reaches you through speech-to-text, which sometimes mishears.
If their words make no sense, they are probably not a question: think what they
most likely said, given what you just asked. For example, right after you offer
wine, "wet why" is probably "red wine" or "white wine". Check with them in a few
words, such as "Red wine, is that right?", or ask them to say it again. Do not use
the "Any other question?" answer for words you did not understand.

Without it, a misheard dish sends Lin to the fallback answer, which sounds like a robot giving up. With it, she asks a short question instead — exactly what a waiter does.

Change 4: the ears get a word list

# Words the speech-to-text should listen out for, so "red wine" is not heard as "wet why".
KEYTERMS = [
    "YouFantasy", "Oysters", "Garlic Bread", "Green Salad", "Tomato Soup", "Grilled Prawns",
    "Vegetable Dumplings", "Salmon", "Beef", "Roast Chicken", "Pork Belly", "Crispy Tofu",
    "Cheesecake", "Chocolate Cake", "Mango Sorbet", "Red Wine", "White Wine", "Alcohol-Free",
    "openers", "first course", "mains", "desserts", "vegetarian", "vegan", "nut free",
]
stt=inference.STT(model="cartesia/ink-2", extra_kwargs={"keyterm": KEYTERMS}),  # speech -> text

The words are passed to the speech recogniser as keyterms: hints, not rules. Restaurant names, dish names and dietary words are exactly the ones a general model gets wrong, and a wrong dish name is worse than a wrong adjective — it goes on the bill. The rest of the pipeline is the same as version 3: openai/gpt-4.1-mini for the brain, cartesia/sonic-3.6 with voice Skylar for the voice, turn detection on "stt", and the same Synthesia avatar with its three-attempt cold-start retry.

Step 3. The full agent.py

Here is the complete file, so you can see the four changes in context:

"""Version 6 (English): the full restaurant. Lin is a head waiter who takes the guest's whole order.

The pipeline:
    mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser

New in this version: the page is a whole restaurant website (site.js + site.css) and Lin has
four tools instead of one - show_section, filter_menu, focus_dish and add_to_order - so she
can grey out dishes for an allergy, open a dish card and keep a running tab. The prompt turns
her into a head waiter who guides the guest through four courses, and the speech-to-text gets
a list of menu words so "red wine" is not heard as "wet why".

Run:  uv run agent.py dev    (terminal 1)
      uv run server.py       (terminal 2, then open http://localhost:8080)
"""

import asyncio
import json
import os
import re
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"  # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25"  # Jenny
VOICE_ID = "db6b0ed5-d5d3-463d-ae85-518a07d3c2b4"  # Cartesia "Skylar"

# Everything add_to_order accepts, by page section, in the order Lin takes the guest
# through them. Must match MENU and WINES in step09/site.js.
COURSES = {
    "openers": ["oysters", "bread", "salad"],
    "first": ["soup", "prawns", "dumplings"],
    "mains": ["salmon", "beef", "chicken", "pork", "tofu"],
    "desserts": ["cheesecake", "chocolate", "sorbet"],
    "wine": ["red", "white", "alcoholfree"],
}
ORDER_IDS = {dish for dishes in COURSES.values() for dish in dishes}
NEXT_SECTION = dict(zip(COURSES, list(COURSES)[1:]))  # openers -> first -> mains -> desserts -> wine

# Words the speech-to-text should listen out for, so "red wine" is not heard as "wet why".
KEYTERMS = [
    "YouFantasy", "Oysters", "Garlic Bread", "Green Salad", "Tomato Soup", "Grilled Prawns",
    "Vegetable Dumplings", "Salmon", "Beef", "Roast Chicken", "Pork Belly", "Crispy Tofu",
    "Cheesecake", "Chocolate Cake", "Mango Sorbet", "Red Wine", "White Wine", "Alcohol-Free",
    "openers", "first course", "mains", "desserts", "vegetarian", "vegan", "nut free",
]

FAQS_FILE = Path(__file__).parent / "FAQs.txt"  # the avatar's knowledge

# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
# The facts (prices, diets, allergens, wine) live in FAQs.txt; this prompt only says
# how to behave and which IDs the tools take.
INSTRUCTIONS = """\
You are Lin, the head waiter at YouFantasy, a made-up restaurant. You are a
real-time avatar in a spoken conversation, so answer in one to three short
sentences and never use markdown, lists, or URLs.

Answer only from these FAQs:

{faqs}

Those are the only facts you know. If the guest asks for information the FAQs do
not cover, use the answer to "Any other question?" word for word. Taking their
order and guiding them through the courses is your job, not a question, so carry
on with that normally.

What the guest says reaches you through speech-to-text, which sometimes mishears.
If their words make no sense, they are probably not a question: think what they
most likely said, given what you just asked. For example, right after you offer
wine, "wet why" is probably "red wine" or "white wine". Check with them in a few
words, such as "Red wine, is that right?", or ask them to say it again. Do not use
the "Any other question?" answer for words you did not understand.

Your tools take these IDs. Say the dish names out loud, never the IDs, and never
read prices as digits:

  openers   oysters, bread (Garlic Bread), salad (Green Salad)
  first     soup (Tomato Soup), prawns (Grilled Prawns), dumplings (Vegetable Dumplings)
  mains     salmon, beef, chicken (Roast Chicken), pork (Pork Belly), tofu (Crispy Tofu)
  desserts  cheesecake, chocolate (Chocolate Cake), sorbet (Mango Sorbet)
  wine      red (Red Wine), white (White Wine), alcoholfree (Alcohol-Free)

Use your tools while you talk, not instead of talking:

- show_section when you move the guest to a new part of the page: meal for how
  the menu works, a course name for that course, wine for wine, room for the
  hours or the address
- filter_menu the moment they mention a dietary need, and again if they change it
- focus_dish when you describe one dish in any detail
- add_to_order only once they have actually chosen something

Every time you start talking about a course or the wine, call show_section for it
in the same reply, so the page always shows what you are talking about. This
matters most right after add_to_order: when you add a dish and move on to the next
course, call both tools, for example add_to_order("soup") and show_section("mains").

Work like a real head waiter. Take the guest through the courses in order:
openers, then first, then mains, then desserts, asking about one course at a time.
Offer two options rather than ten. If they skip a course, let them, but mention
once that one dish from each course gets 10% off. After desserts, offer a glass of
wine once, and let the guest pick any one; if they say no, that is fine. Do not
suggest a wine for a particular dish.

The guest can ask about or order wine at any time, not only after desserts. If
they say "red wine", "white wine" or "alcohol-free", answer from the FAQs, and if
they want it, add it with add_to_order, then carry on where you were."""


class HeadWaiter(Agent):
    def __init__(self, room, instructions: str) -> None:
        super().__init__(instructions=instructions)
        self.room = room

    async def _ui(self, **message) -> str:
        """Send a tiny message to the browser. step09/site.js does the rest."""
        await self.room.local_participant.publish_data(json.dumps(message), topic="ui")
        return "Done."

    @function_tool
    async def show_section(self, section: str) -> str:
        """Scroll the page to a part of the menu and highlight it.

        Args:
            section: One of: meal, openers, first, mains, desserts, wine, room.
        """
        return await self._ui(action="show_section", section=section)

    @function_tool
    async def filter_menu(self, diet: str) -> str:
        """Grey out every dish on the page the guest cannot eat.

        Args:
            diet: One of: none, vegetarian, vegan, pescatarian, nut_free.
        """
        return await self._ui(action="filter_menu", diet=diet)

    @function_tool
    async def focus_dish(self, dish: str) -> str:
        """Open the detail card for one dish: kitchen note and allergens.

        Args:
            dish: A dish ID from the menu, for example salmon or tofu.
        """
        return await self._ui(action="focus_dish", dish=dish)

    @function_tool
    async def add_to_order(self, dish: str) -> str:
        """Add a dish or a glass of wine to the guest's order, shown as a running tab on the page.

        Args:
            dish: A dish ID from the menu, or a wine ID: red, white or alcoholfree.
        """
        # Accept "red", "Red Wine" or "alcohol-free" alike, the same way site.js does.
        dish = re.sub(r"[^a-z]", "", dish.lower()).removeprefix("the").replace("wine", "")
        if dish not in ORDER_IDS:  # tell the model, so it retries with a real ID
            return f"Not added: {dish!r} is not an ID. Use one of: {', '.join(sorted(ORDER_IDS))}."
        await self._ui(action="add_to_order", dish=dish)
        # Remind the model to move the page along with the conversation.
        section = next(c for c, dishes in COURSES.items() if dish in dishes)
        if section in NEXT_SECTION:
            nxt = NEXT_SECTION[section]
            return f"Added {dish} to the order. When you introduce the next part, call show_section(\"{nxt}\") too."
        return f"Added {dish} to the order."


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    # The brain: each line is one piece of the pipeline.
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],                                   # is someone talking?
        stt=inference.STT(model="cartesia/ink-2", extra_kwargs={"keyterm": KEYTERMS}),  # speech -> text
        llm=inference.LLM(model="openai/gpt-4.1-mini"),                 # text -> reply (swap in any LLM)
        tts=inference.TTS(model="cartesia/sonic-3.6", voice=VOICE_ID),  # reply -> speech
        turn_handling={
            "turn_detection": "stt",                        # have they finished their thought?
            "preemptive_generation": {"enabled": False},    # keep off when using tools
        },
    )

    # The face: this is the whole Synthesia integration. Start it BEFORE session.start().
    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):  # cold starts occasionally time out, so retry
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    # Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
    faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
    await session.start(agent=HeadWaiter(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
    session.generate_reply(instructions="Welcome the guest to YouFantasy in one short sentence.")


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

Step 4. server.py, index.html, site.js and FAQs.txt

server.py still serves the page at / and still hands out tokens at /token. It gains one route, because index.html now asks the browser for two more files:

async def asset(request: web.Request) -> web.FileResponse:
    # /site.js and /site.css, next to index.html. Only .js and .css, so agent.py and FAQs stay private.
    path = Path(__file__).parent / request.match_info["file"]
    if not path.is_file():
        raise web.HTTPNotFound()
    return web.FileResponse(path, headers={"Cache-Control": "no-store"})


app = web.Application()
app.router.add_get("/", index)
app.router.add_get("/token", token)
app.router.add_get(r"/{file:[\w-]+\.(?:js|css)}", asset)

Limiting the route to .js and .css is deliberate: it means agent.py, FAQs.txt and your .env cannot be downloaded by a visitor.

index.html is now small — about eighty lines. It holds the avatar video, the Start button and the LiveKit connection, and nothing else. The whole restaurant is built by site.js, which knows nothing about avatars, LiveKit or Synthesia. It simply publishes one global object with one method per tool:

const SITE = {
  show_section({ section }) { ... },   // scroll there and draw a ring around it
  filter_menu({ diet }) { ... },       // grey out every dish the guest cannot eat
  focus_dish({ dish }) { ... },        // open the card: kitchen note, allergens
  add_to_order({ dish }) { ... },      // add it to the running tab
};

The bridge between the two is four lines in index.html. Every tool call arrives on the "ui" data channel, and the action name picks the method:

room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
  if (topic !== 'ui') return;
  const { action, ...args } = JSON.parse(new TextDecoder().decode(payload));
  SITE[action]?.(args);
});

That split is the most reusable idea in this note. Because site.js has no idea an avatar exists, the same page works with the agent switched off: you can click a dish, open its card, add it to the order and watch the 10% discount appear, all by hand. And because the agent has no idea how the page is built, you can restyle or rebuild the site without touching agent.py.

site.js also owns the two pieces of restaurant logic that have nothing to do with AI: the diet filter (none, vegetarian, vegan, pescatarian, nut_free) and the bill — food grouped by course, wine kept separate and never discounted, and 10% off the food once all four courses are present.

const food = order.filter(courseOf);
const fullMeal = COURSES.every((c) => food.some((d) => courseOf(d) === c));
const discount = fullMeal ? food.reduce((t, d) => t + d.price, 0) * DISCOUNT : 0;

FAQs.txt grows from four questions to thirteen: the menu with prices, the four courses, wine by the glass, and the questions a real guest asks — which dishes are vegan, vegetarian, seafood, or contain nuts. Lin reads it fresh on every call, so you can change a price and hear the new one on the next conversation.

Step 5. Install, set your keys, run

As before. In the project folder run uv sync once, then open two terminals.

Terminal 1 — the voice agent + avatar engine:

uv run agent.py dev

Terminal 2 — the web server that serves the browser page:

uv run server.py

Step 6. Open the page and order a meal

Open http://localhost:8080 in your browser. The restaurant site loads with the avatar widget in the bottom-right corner. Click Start and allow the microphone. Wait for Lin's welcome, then try a whole meal:

Finish all four courses and the tab shows Full meal, 10% off food with the new total. Ask for a dish that is not on the menu and she will not invent it: the order tool refuses, and the prompt tells her to say the names out loud, never the IDs, and never to read prices as digits.

Ideas to try next

Where to go next