English 繁體中文 简体中文

How to use LiveKit and Synthesia Avatar with webpage (version 5)-Cantonese?

Howard Lam, PhD · AI Notes · November 6, 2026 · 8 min read

In an earlier note, Lin the restaurant host moved into a real web page: when she answered a question, the page scrolled to the part she was talking about. In the Putonghua version, she learned to speak Mandarin Chinese.

This note makes the same leap in a different direction: Lin now speaks Cantonese — the colloquial Hong Kong kind, not standard written Chinese. You ask “你哋幾點開門?” and she answers the way a friendly waiter would, in traditional characters, while the page scrolls to the opening hours. Her knowledge file is still in English, and so is the web page; the AI translates on the fly.

Putonghua vs. Cantonese at a glance

Version 3 (Putonghua)Version 5 (Cantonese)
Conversation languagePutonghua (Mandarin Chinese)Colloquial Hong Kong Cantonese
Ears (speech to text)Cartesia ink-whisper, set to ChineseSpeechmatics linden-1, set to Cantonese
Voice (text to speech)Cartesia sonic-3, voice “Jing”Inworld TTS 2, voice “Xinyi”
Knowing when you finish speakingSilence detection (VAD)STT model tells it, on its servers
CharactersSimplifiedTraditional, with spoken words like 係, 唔, 喺
Knowledge file FAQs.txtEnglish — translated on the flyEnglish — translated on the fly
Page scrolls and highlightsYes (show_section tool)Yes (show_section tool)
New accounts or keysNoneNone

What you need before you start

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds agent.py, server.py, index.html, FAQs.txt, pyproject.toml, and README.md. Add your own .env file with the same five keys as before.

Step 1. Set up the project folder

Five of the six files — server.py, FAQs.txt, pyproject.toml, and (apart from one line) index.html — are the same as before, so if you already have one of the earlier folders you can copy it and replace only agent.py.

The .env file with your five secret keys also stays exactly the same. Only agent.py really changes, and all the changes are again about language.

Step 2. Look at agent.py — what changed

There are four changes, and the first one forces the second.

Change 1: new ears that understand Cantonese

stt=inference.STT(model="speechmatics/linden-1", language="yue"),

The speech-to-text model — Lin's ears — changes from Cartesia's ink-whisper to Speechmatics linden-1, with language="yue" (yue is the code for Cantonese). Cartesia's models handle Putonghua but not Cantonese, so this is a different provider.

You do not need a Speechmatics account: the model runs through LiveKit Inference, the same service that already provided the previous ears and voice, billed through your LiveKit project.

Change 2: turn detection goes back to the STT model

turn_handling={
    "turn_detection": "stt",                        # have they finished? (Speechmatics decides, on its servers)
    "preemptive_generation": {"enabled": False},    # keep off when using tools
},

In the Putonghua version we had to fall back to "vad" — guessing from silence — because ink-whisper has no turn-detection signal. Speechmatics does report the end of a turn, so we can switch back to "stt": Lin now waits until the recogniser itself says you have finished, which handles the rushed, half-swallowed endings of spoken Cantonese better than a fixed pause.

Change 3: a new Cantonese voice

VOICE_ID = "Xinyi"  # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)
tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"),

The voice changes from Cartesia's Mandarin Jing to Inworld's Xinyi, with language="yue". Note that Inworld voices are named by a plain name, not a long ID like Cartesia's — you can try Xiaoyin, Jing, or Yichen (male) by changing that one line. The avatar still speaks through Synthesia exactly as before.

Change 4: the prompt asks for colloquial Cantonese

The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.

This is longer than the Putonghua line, for a reason. Without help, a Chinese-speaking LLM drifts towards standard written Chinese, which sounds stiff when spoken aloud and is not what a Hong Kong waiter would say. Naming the six everyday words and asking for traditional characters keeps Lin in spoken Cantonese.

The address rule is there because a place name can be read aloud in several ways; spelling it out makes sure the voice says 九龍塘達之路 the way a Hong Kong listener expects. The fallback line and the welcome message also change:

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.
session.generate_reply(
    instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
)

Everything else — the show_section tool, the retry loop for the avatar, reading FAQs.txt fresh on every call — is unchanged.

Step 3. The full agent.py

Here is the complete file, so you can see the four changes in context:

"""Version 5 (Cantonese): Lin speaks Cantonese. The guest asks in Cantonese and Lin answers in Cantonese.

The pipeline:
    mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
    VAD: Voice Activity Detection, STT: Speech-to-Text, LLM: Large Language Model, TTS: Text-to-Speech

New in this version: the ears (Speechmatics) and the voice (Inworld) change to models
that handle Cantonese, and the prompt tells Lin to reply in colloquial Hong Kong
Cantonese. Both still run through LiveKit Inference, so no new API key is needed.
FAQs.txt, the prompt and the web page stay in English: the LLM translates on the fly.

Run:  uv run agent.py dev    (terminal 1)
      uv run server.py       (terminal 2, then open http://localhost:8080)
"""

import asyncio
import json
import os
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"  # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25"  # Jenny
VOICE_ID = "Xinyi"  # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)

FAQS_FILE = Path(__file__).parent / "FAQs.txt"  # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"}       # must match the section ids in index.html

# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.

The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.

Answer only from these FAQs:

{faqs}

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.

The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:

- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")

Do not call show_section for any other question."""


class Host(Agent):
    def __init__(self, room, instructions: str) -> None:
        super().__init__(instructions=instructions)
        self.room = room  # needed to send messages to the browser

    @function_tool
    async def show_section(self, section: str) -> str:
        """Scroll the guest's web page to one section and highlight it.

        Args:
            section: One of: hours, mains, location.
        """
        if section not in SECTIONS:  # tell the model, so it retries with a real name
            return f"No section {section!r}. Use one of: hours, mains, location."
        message = json.dumps({"action": "show_section", "section": section})
        await self.room.local_participant.publish_data(message, topic="ui")  # index.html listens on "ui"
        return "Done."


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    # The brain: each line is one piece of the pipeline.
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],                                              # is someone talking?
        stt=inference.STT(model="speechmatics/linden-1", language="yue"),          # Cantonese speech -> text
        llm=inference.LLM(model="openai/gpt-4.1-mini"),                            # text -> reply (+ tool calls)
        tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"),  # Cantonese reply -> speech
        turn_handling={
            "turn_detection": "stt",                        # have they finished? (Speechmatics decides, on its servers)
            "preemptive_generation": {"enabled": False},    # keep off when using tools
        },
    )

    # The face. Start it BEFORE session.start().
    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):  # cold starts occasionally time out, so retry
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    # Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
    faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
    await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
    session.generate_reply(
        instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
    )


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

Step 4. server.py, index.html and FAQs.txt

server.py is exactly the same file as before: it still serves the page at / and still hands out tokens at /token.

index.html keeps the same sections (hours, mains, location), the same gold highlight ring and the same listener on the "ui" data channel. Only two lines change: the page title and the one-line introduction, which now invites visitors to ask in Cantonese:

<p>Ask Lin, our host, in Cantonese about our hours, our main courses or where to find us.</p>

The page itself stays in English, which makes the demo easy to follow: Lin speaks Cantonese while the English page scrolls along with her.

FAQs.txt is untouched — the same four English Q&A pairs, translated by the LLM at call time.

Step 5. Install, set your keys, run

Identical to the earlier versions. In the project folder run uv sync once, then open two terminals.

Terminal 1 — the voice agent + avatar engine:

uv run agent.py dev

Terminal 2 — the web server that serves the browser page:

uv run server.py

Step 6. Open the page and talk to Lin in Cantonese

Open http://localhost:8080 in your browser. The restaurant page loads with the avatar widget in the bottom-right corner. Click Start and allow the microphone.

Wait for Lin's welcome, then ask in Cantonese, for example:

Listen for the giveaway words 係, 唔, 喺, 嘅, 咗 and 啲: they tell you she is speaking Cantonese, not reading standard written Chinese aloud.

Ideas to try next

Where to go next