How to use LiveKit and Synthesia Avatar with webpage (version 4)-PuTongHua?
In version 2, Lin the restaurant host answered your spoken questions with knowledge from a plain text file. In version 3, she moved into a real web page: when she answered, the page scrolled to the part she was talking about.
Version 4 makes one more change: Lin now speaks Putonghua (Mandarin Chinese). You ask a question in Putonghua, she answers in Putonghua with a Mandarin voice, and the page still scrolls as she speaks. Here is the interesting part: her knowledge file is still in English, and so is the web page — the AI model translates everything on the fly.
Version 3 vs. version 4 at a glance
| Version 3 | Version 4 | |
|---|---|---|
| Conversation language | English | Putonghua (Mandarin Chinese) |
| Ears (speech to text) | Cartesia ink-2 | Cartesia ink-whisper, set to Chinese |
| Voice (text to speech) | Cartesia sonic-3.6, voice “Skylar” | Cartesia sonic-3, voice “Jing” (Mandarin) |
| Knowing when you finish speaking | STT model tells it | Silence detection (VAD) |
| Knowledge file FAQs.txt | English | English — translated on the fly |
| Web page text | English | English (unchanged) |
| Page scrolls and highlights | Yes (show_section tool) | Yes (show_section tool) |
What you need before you start
- Everything version 3 needed — see How to use LiveKit and Synthesia Avatar with webpage (version 3)? for the full list.
- No new accounts or keys: the same LiveKit and Synthesia credentials work.
Running it on your own computer
Download the zip file and
unzip it to a folder. The zip holds agent.py, server.py,
index.html, FAQs.txt, pyproject.toml, and
README.md. Add your own .env file with the same five keys as
version 3.
Step 1. Set up the project folder
Four of the six files — server.py,
index.html, FAQs.txt and pyproject.toml —
are identical to version 3, so if you already have that folder you can copy it and
replace only agent.py.
The .env file with your five secret keys stays exactly the same as in
version 3. The packages in pyproject.toml are also unchanged — the
Chinese-speaking models come from the same Cartesia service that LiveKit already
connects to.
Step 2. Look at agent.py — what changed
The file is almost the same as version 3. Only four things changed, and all of them are about language.
Change 1: new ears that understand Chinese
stt=inference.STT(model="cartesia/ink-whisper", language="zh"),
The speech-to-text model — Lin's ears — changes from
cartesia/ink-2 to cartesia/ink-whisper, with
language="zh" (zh is the code for Chinese). This model listens to
Putonghua and writes down what you said, in Chinese characters.
The switch is forced: ink-2 only understands English, so it cannot be
kept for Putonghua. ink-whisper is the multilingual model that does.
Change 2: a new Mandarin voice
VOICE_ID = "6eb8965c-e295-47bd-a9e4-3eeebb3abcff" # Cartesia "Jing", a Mandarin female voice
tts=inference.TTS(model="cartesia/sonic-3", voice=VOICE_ID, language="zh"),
The text-to-speech voice changes to Jing, a Mandarin female
voice, again with language="zh". The avatar still speaks through
Synthesia exactly as before — only the sound of her voice is new.
Change 3: a different way to notice you have finished speaking
turn_handling={
"turn_detection": "vad", # have they finished? (ink-whisper can't tell, so use silence)
"preemptive_generation": {"enabled": False}, # keep off when using tools
},
In version 3 the STT model itself reported when you had finished a sentence. The new
ink-whisper model cannot do that — unlike ink-2, it has no turn-detection
signal — so we switch turn_detection to "vad": the
program now decides you have finished simply by noticing a moment of silence.
Change 4: two new lines in the prompt
The guest speaks Putonghua (Mandarin Chinese). Always reply in Putonghua, written in
simplified Chinese characters, even though the FAQs below are in English.
This is the heart of version 4. The FAQs in FAQs.txt stay in English, so
you keep editing them the same way as before. The LLM reads the English answer and
says it in Putonghua. The fallback line also changes slightly:
Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Putonghua.
And the welcome message is now requested in Putonghua too:
session.generate_reply(
instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Putonghua."
)
Everything else — the show_section tool, the retry loop for the
avatar, reading FAQs.txt fresh on every call — is identical to version 3.
Step 3. The full agent.py
Here is the complete file, so you can see the four changes in context:
"""Version 4: Lin speaks Putonghua. The guest asks in Putonghua and Lin answers in Putonghua.
The pipeline:
mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
VAD: Voice Activity Detection, STT: Speech-to-Text, LLM: Large Language Model, TTS: Text-to-Speech
New in version 4: the ears and the voice change to models that handle Chinese, and
the prompt tells Lin to reply in Putonghua. FAQs.txt, the prompt and the web page
stay in English: the LLM translates on the fly.
Run: uv run agent.py dev (terminal 1)
uv run server.py (terminal 2, then open http://localhost:8080)
"""
import asyncio
import json
import os
from pathlib import Path
from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia
load_dotenv(dotenv_path=Path(__file__).parent / ".env")
AGENT_NAME = "avatar-demo" # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25" # Jenny
VOICE_ID = "6eb8965c-e295-47bd-a9e4-3eeebb3abcff" # Cartesia "Jing", a Mandarin female voice
FAQS_FILE = Path(__file__).parent / "FAQs.txt" # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"} # must match the section ids in index.html
# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.
The guest speaks Putonghua (Mandarin Chinese). Always reply in Putonghua, written in
simplified Chinese characters, even though the FAQs below are in English.
Answer only from these FAQs:
{faqs}
Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Putonghua.
The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:
- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")
Do not call show_section for any other question."""
class Host(Agent):
def __init__(self, room, instructions: str) -> None:
super().__init__(instructions=instructions)
self.room = room # needed to send messages to the browser
@function_tool
async def show_section(self, section: str) -> str:
"""Scroll the guest's web page to one section and highlight it.
Args:
section: One of: hours, mains, location.
"""
if section not in SECTIONS: # tell the model, so it retries with a real name
return f"No section {section!r}. Use one of: hours, mains, location."
message = json.dumps({"action": "show_section", "section": section})
await self.room.local_participant.publish_data(message, topic="ui") # index.html listens on "ui"
return "Done."
def prewarm(proc) -> None:
proc.userdata["vad"] = silero.VAD.load()
async def entrypoint(ctx: JobContext) -> None:
await ctx.connect()
# The brain: each line is one piece of the pipeline.
session = AgentSession(
vad=ctx.proc.userdata["vad"], # is someone talking?
stt=inference.STT(model="cartesia/ink-whisper", language="zh"), # Chinese speech -> text
llm=inference.LLM(model="openai/gpt-4.1-mini"), # text -> reply (+ tool calls)
tts=inference.TTS(model="cartesia/sonic-3", voice=VOICE_ID, language="zh"), # Chinese reply -> speech
turn_handling={
"turn_detection": "vad", # have they finished? (ink-whisper can't tell, so use silence)
"preemptive_generation": {"enabled": False}, # keep off when using tools
},
)
# The face. Start it BEFORE session.start().
avatar = synthesia.AvatarSession(
synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
join_timeout=60.0,
)
for attempt in range(3): # cold starts occasionally time out, so retry
try:
await avatar.start(session, room=ctx.room)
break
except synthesia.SynthesiaError as e:
if not e.retryable or attempt == 2:
raise
await asyncio.sleep(2)
# Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
session.generate_reply(
instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Putonghua."
)
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))
Step 4. server.py, index.html and FAQs.txt — unchanged
server.py is exactly the same file as in version 3: it still serves the
page at / and still hands out tokens at /token.
index.html keeps the same sections (hours,
mains, location), the same gold highlight ring and the same
listener on the "ui" data channel. Only the one-line introduction now
invites visitors to ask in Putonghua:
<p>Ask Lin, our host, in Putonghua about our hours, our main courses or where to find us.</p>
The page itself stays in English, which makes the demo easy to follow: Lin speaks Chinese while the English page scrolls along with her.
FAQs.txt is also untouched — the same four English Q&A pairs,
translated by the LLM at call time.
Step 5. Install, set your keys, run
Identical to version 3. In the project folder run uv sync once, then
open two terminals:
uv run agent.py dev
and in the second:
uv run server.py
Step 6. Open the page and talk to Lin in Putonghua
Open http://localhost:8080 in your browser. The restaurant page loads
with the avatar widget in the bottom-right corner. Click Start and
allow the microphone.
Wait for Lin's welcome, then ask in Putonghua, for example:
- “你们几点开门?” — she answers with the opening hours and the page jumps to Opening hours.
- “你们在哪里?” — she answers with the address and the page jumps to Where to find us.
- “你们有停车位吗?” — that is not in the FAQs, so she gives the “Any other question?” answer, translated into Putonghua, and the page stays still.
Ideas to try next
- Add a new Q&A pair to FAQs.txt — still in English — save it, and refresh the page. Lin answers with the new knowledge, in Putonghua, on the next call.
- Rename a section in index.html and update
SECTIONSand the rule book in agent.py to match. - Try another language by changing the
languagecode and the voice — for example"yue"for Cantonese with a matching Cartesia voice.
Where to go next
- How to use LiveKit and Synthesia Avatar with webpage (version 3)? — the page-scrolling avatar this version builds on.
- How to use LiveKit and Synthesia Avatar (version 2)? — the same avatar with its knowledge in a plain text file.
- How to ask a LiveKit voice agent about the weather? — the first note about giving the agent tools to call.
- How to install Homebrew on macOS? — if you still need
uvon a Mac.