How to use LiveKit and Synthesia Avatar with webpage (version 5)-Cantonese?
In an earlier note, Lin the restaurant host moved into a real web page: when she answered a question, the page scrolled to the part she was talking about. In the Putonghua version, she learned to speak Mandarin Chinese.
This note makes the same leap in a different direction: Lin now speaks Cantonese — the colloquial Hong Kong kind, not standard written Chinese. You ask “你哋幾點開門?” and she answers the way a friendly waiter would, in traditional characters, while the page scrolls to the opening hours. Her knowledge file is still in English, and so is the web page; the AI translates on the fly.
Putonghua vs. Cantonese at a glance
| Version 3 (Putonghua) | Version 5 (Cantonese) | |
|---|---|---|
| Conversation language | Putonghua (Mandarin Chinese) | Colloquial Hong Kong Cantonese |
| Ears (speech to text) | Cartesia ink-whisper, set to Chinese | Speechmatics linden-1, set to Cantonese |
| Voice (text to speech) | Cartesia sonic-3, voice “Jing” | Inworld TTS 2, voice “Xinyi” |
| Knowing when you finish speaking | Silence detection (VAD) | STT model tells it, on its servers |
| Characters | Simplified | Traditional, with spoken words like 係, 唔, 喺 |
| Knowledge file FAQs.txt | English — translated on the fly | English — translated on the fly |
| Page scrolls and highlights | Yes (show_section tool) | Yes (show_section tool) |
| New accounts or keys | None | None |
What you need before you start
- Everything the earlier versions needed — see How to use LiveKit and Synthesia Avatar with webpage (version 3)? for the full list.
- No new accounts, keys or packages: both the Cantonese ears and the Cantonese voice run through LiveKit Inference, which your existing LiveKit project already uses.
Running it on your own computer
Download the zip file and
unzip it to a folder. The zip holds agent.py, server.py,
index.html, FAQs.txt, pyproject.toml, and
README.md. Add your own .env file with the same five keys as
before.
Step 1. Set up the project folder
Five of the six files — server.py, FAQs.txt,
pyproject.toml, and (apart from one line) index.html —
are the same as before, so if you already have one of the earlier folders you can copy
it and replace only agent.py.
The .env file with your five secret keys also stays exactly the same. Only
agent.py really changes, and all the changes are again about language.
Step 2. Look at agent.py — what changed
There are four changes, and the first one forces the second.
Change 1: new ears that understand Cantonese
stt=inference.STT(model="speechmatics/linden-1", language="yue"),
The speech-to-text model — Lin's ears — changes from
Cartesia's ink-whisper to Speechmatics linden-1, with
language="yue" (yue is the code for Cantonese). Cartesia's models handle
Putonghua but not Cantonese, so this is a different provider.
You do not need a Speechmatics account: the model runs through LiveKit Inference, the same service that already provided the previous ears and voice, billed through your LiveKit project.
Change 2: turn detection goes back to the STT model
turn_handling={
"turn_detection": "stt", # have they finished? (Speechmatics decides, on its servers)
"preemptive_generation": {"enabled": False}, # keep off when using tools
},
In the Putonghua version we had to fall back to "vad" — guessing from
silence — because ink-whisper has no turn-detection signal. Speechmatics does
report the end of a turn, so we can switch back to "stt": Lin now waits
until the recogniser itself says you have finished, which handles the rushed,
half-swallowed endings of spoken Cantonese better than a fixed pause.
Change 3: a new Cantonese voice
VOICE_ID = "Xinyi" # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)
tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"),
The voice changes from Cartesia's Mandarin Jing to Inworld's
Xinyi, with language="yue". Note that Inworld voices are named by
a plain name, not a long ID like Cartesia's — you can try
Xiaoyin, Jing, or Yichen (male) by changing that
one line. The avatar still speaks through Synthesia exactly as before.
Change 4: the prompt asks for colloquial Cantonese
The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.
This is longer than the Putonghua line, for a reason. Without help, a Chinese-speaking LLM drifts towards standard written Chinese, which sounds stiff when spoken aloud and is not what a Hong Kong waiter would say. Naming the six everyday words and asking for traditional characters keeps Lin in spoken Cantonese.
The address rule is there because a place name can be read aloud in several ways; spelling it out makes sure the voice says 九龍塘達之路 the way a Hong Kong listener expects. The fallback line and the welcome message also change:
Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.
session.generate_reply(
instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
)
Everything else — the show_section tool, the retry loop for the
avatar, reading FAQs.txt fresh on every call — is unchanged.
Step 3. The full agent.py
Here is the complete file, so you can see the four changes in context:
"""Version 5 (Cantonese): Lin speaks Cantonese. The guest asks in Cantonese and Lin answers in Cantonese.
The pipeline:
mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
VAD: Voice Activity Detection, STT: Speech-to-Text, LLM: Large Language Model, TTS: Text-to-Speech
New in this version: the ears (Speechmatics) and the voice (Inworld) change to models
that handle Cantonese, and the prompt tells Lin to reply in colloquial Hong Kong
Cantonese. Both still run through LiveKit Inference, so no new API key is needed.
FAQs.txt, the prompt and the web page stay in English: the LLM translates on the fly.
Run: uv run agent.py dev (terminal 1)
uv run server.py (terminal 2, then open http://localhost:8080)
"""
import asyncio
import json
import os
from pathlib import Path
from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia
load_dotenv(dotenv_path=Path(__file__).parent / ".env")
AGENT_NAME = "avatar-demo" # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25" # Jenny
VOICE_ID = "Xinyi" # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)
FAQS_FILE = Path(__file__).parent / "FAQs.txt" # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"} # must match the section ids in index.html
# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.
The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.
Answer only from these FAQs:
{faqs}
Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.
The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:
- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")
Do not call show_section for any other question."""
class Host(Agent):
def __init__(self, room, instructions: str) -> None:
super().__init__(instructions=instructions)
self.room = room # needed to send messages to the browser
@function_tool
async def show_section(self, section: str) -> str:
"""Scroll the guest's web page to one section and highlight it.
Args:
section: One of: hours, mains, location.
"""
if section not in SECTIONS: # tell the model, so it retries with a real name
return f"No section {section!r}. Use one of: hours, mains, location."
message = json.dumps({"action": "show_section", "section": section})
await self.room.local_participant.publish_data(message, topic="ui") # index.html listens on "ui"
return "Done."
def prewarm(proc) -> None:
proc.userdata["vad"] = silero.VAD.load()
async def entrypoint(ctx: JobContext) -> None:
await ctx.connect()
# The brain: each line is one piece of the pipeline.
session = AgentSession(
vad=ctx.proc.userdata["vad"], # is someone talking?
stt=inference.STT(model="speechmatics/linden-1", language="yue"), # Cantonese speech -> text
llm=inference.LLM(model="openai/gpt-4.1-mini"), # text -> reply (+ tool calls)
tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"), # Cantonese reply -> speech
turn_handling={
"turn_detection": "stt", # have they finished? (Speechmatics decides, on its servers)
"preemptive_generation": {"enabled": False}, # keep off when using tools
},
)
# The face. Start it BEFORE session.start().
avatar = synthesia.AvatarSession(
synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
join_timeout=60.0,
)
for attempt in range(3): # cold starts occasionally time out, so retry
try:
await avatar.start(session, room=ctx.room)
break
except synthesia.SynthesiaError as e:
if not e.retryable or attempt == 2:
raise
await asyncio.sleep(2)
# Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
session.generate_reply(
instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
)
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))
Step 4. server.py, index.html and FAQs.txt
server.py is exactly the same file as before: it still serves the page at
/ and still hands out tokens at /token.
index.html keeps the same sections (hours,
mains, location), the same gold highlight ring and the same
listener on the "ui" data channel. Only two lines change: the page title
and the one-line introduction, which now invites visitors to ask in Cantonese:
<p>Ask Lin, our host, in Cantonese about our hours, our main courses or where to find us.</p>
The page itself stays in English, which makes the demo easy to follow: Lin speaks Cantonese while the English page scrolls along with her.
FAQs.txt is untouched — the same four English Q&A pairs,
translated by the LLM at call time.
Step 5. Install, set your keys, run
Identical to the earlier versions. In the project folder run uv sync once,
then open two terminals.
Terminal 1 — the voice agent + avatar engine:
uv run agent.py dev
Terminal 2 — the web server that serves the browser page:
uv run server.py
Step 6. Open the page and talk to Lin in Cantonese
Open http://localhost:8080 in your browser. The restaurant page loads with
the avatar widget in the bottom-right corner. Click Start and allow
the microphone.
Wait for Lin's welcome, then ask in Cantonese, for example:
- “你哋幾點開門?” — she answers with the opening hours and the page jumps to Opening hours.
- “你哋喺邊度?” — she answers with the address, saying 九龍塘達之路, and the page jumps to Where to find us.
- “有冇泊車位?” — that is not in the FAQs, so she gives the “Any other question?” answer, in Cantonese, and the page stays still.
Listen for the giveaway words 係, 唔, 喺, 嘅, 咗 and 啲: they tell you she is speaking Cantonese, not reading standard written Chinese aloud.
Ideas to try next
- Add a new Q&A pair to FAQs.txt — still in English — save it, and refresh the page. Lin answers with the new knowledge, in Cantonese, on the next call.
- Change
VOICE_IDtoXiaoyin,Jing, orYichen(male) and compare the voices. - Go back to Putonghua by copying the Putonghua version of
agent.py; only four lines differ.
Where to go next
- How to use LiveKit and Synthesia Avatar with webpage (version 4)-PuTongHua? — the Putonghua version this note mirrors.
- How to use LiveKit and Synthesia Avatar with webpage (version 3)? — the page-scrolling avatar this version builds on.
- How to use LiveKit and Synthesia Avatar (version 2)? — the same avatar with its knowledge in a plain text file.
- How to install Homebrew on macOS? — if you still need
uvon a Mac.