English 繁體中文 简体中文

How to use LiveKit and Synthesia Avatar with webpage (version 4)-PuTongHua?

Howard Lam, PhD · AI Notes · November 6, 2026 · 8 min read

In version 2, Lin the restaurant host answered your spoken questions with knowledge from a plain text file. In version 3, she moved into a real web page: when she answered, the page scrolled to the part she was talking about.

Version 4 makes one more change: Lin now speaks Putonghua (Mandarin Chinese). You ask a question in Putonghua, she answers in Putonghua with a Mandarin voice, and the page still scrolls as she speaks. Here is the interesting part: her knowledge file is still in English, and so is the web page — the AI model translates everything on the fly.

Version 3 vs. version 4 at a glance

Version 3Version 4
Conversation languageEnglishPutonghua (Mandarin Chinese)
Ears (speech to text)Cartesia ink-2Cartesia ink-whisper, set to Chinese
Voice (text to speech)Cartesia sonic-3.6, voice “Skylar”Cartesia sonic-3, voice “Jing” (Mandarin)
Knowing when you finish speakingSTT model tells itSilence detection (VAD)
Knowledge file FAQs.txtEnglishEnglish — translated on the fly
Web page textEnglishEnglish (unchanged)
Page scrolls and highlightsYes (show_section tool)Yes (show_section tool)

What you need before you start

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds agent.py, server.py, index.html, FAQs.txt, pyproject.toml, and README.md. Add your own .env file with the same five keys as version 3.

Step 1. Set up the project folder

Four of the six files — server.py, index.html, FAQs.txt and pyproject.toml — are identical to version 3, so if you already have that folder you can copy it and replace only agent.py.

The .env file with your five secret keys stays exactly the same as in version 3. The packages in pyproject.toml are also unchanged — the Chinese-speaking models come from the same Cartesia service that LiveKit already connects to.

Step 2. Look at agent.py — what changed

The file is almost the same as version 3. Only four things changed, and all of them are about language.

Change 1: new ears that understand Chinese

stt=inference.STT(model="cartesia/ink-whisper", language="zh"),

The speech-to-text model — Lin's ears — changes from cartesia/ink-2 to cartesia/ink-whisper, with language="zh" (zh is the code for Chinese). This model listens to Putonghua and writes down what you said, in Chinese characters.

The switch is forced: ink-2 only understands English, so it cannot be kept for Putonghua. ink-whisper is the multilingual model that does.

Change 2: a new Mandarin voice

VOICE_ID = "6eb8965c-e295-47bd-a9e4-3eeebb3abcff"  # Cartesia "Jing", a Mandarin female voice
tts=inference.TTS(model="cartesia/sonic-3", voice=VOICE_ID, language="zh"),

The text-to-speech voice changes to Jing, a Mandarin female voice, again with language="zh". The avatar still speaks through Synthesia exactly as before — only the sound of her voice is new.

Change 3: a different way to notice you have finished speaking

turn_handling={
    "turn_detection": "vad",                        # have they finished? (ink-whisper can't tell, so use silence)
    "preemptive_generation": {"enabled": False},    # keep off when using tools
},

In version 3 the STT model itself reported when you had finished a sentence. The new ink-whisper model cannot do that — unlike ink-2, it has no turn-detection signal — so we switch turn_detection to "vad": the program now decides you have finished simply by noticing a moment of silence.

Change 4: two new lines in the prompt

The guest speaks Putonghua (Mandarin Chinese). Always reply in Putonghua, written in
simplified Chinese characters, even though the FAQs below are in English.

This is the heart of version 4. The FAQs in FAQs.txt stay in English, so you keep editing them the same way as before. The LLM reads the English answer and says it in Putonghua. The fallback line also changes slightly:

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Putonghua.

And the welcome message is now requested in Putonghua too:

session.generate_reply(
    instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Putonghua."
)

Everything else — the show_section tool, the retry loop for the avatar, reading FAQs.txt fresh on every call — is identical to version 3.

Step 3. The full agent.py

Here is the complete file, so you can see the four changes in context:

"""Version 4: Lin speaks Putonghua. The guest asks in Putonghua and Lin answers in Putonghua.

The pipeline:
    mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
    VAD: Voice Activity Detection, STT: Speech-to-Text, LLM: Large Language Model, TTS: Text-to-Speech

New in version 4: the ears and the voice change to models that handle Chinese, and
the prompt tells Lin to reply in Putonghua. FAQs.txt, the prompt and the web page
stay in English: the LLM translates on the fly.

Run:  uv run agent.py dev    (terminal 1)
      uv run server.py       (terminal 2, then open http://localhost:8080)
"""

import asyncio
import json
import os
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"  # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25"  # Jenny
VOICE_ID = "6eb8965c-e295-47bd-a9e4-3eeebb3abcff"  # Cartesia "Jing", a Mandarin female voice

FAQS_FILE = Path(__file__).parent / "FAQs.txt"  # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"}       # must match the section ids in index.html

# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.

The guest speaks Putonghua (Mandarin Chinese). Always reply in Putonghua, written in
simplified Chinese characters, even though the FAQs below are in English.

Answer only from these FAQs:

{faqs}

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Putonghua.

The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:

- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")

Do not call show_section for any other question."""


class Host(Agent):
    def __init__(self, room, instructions: str) -> None:
        super().__init__(instructions=instructions)
        self.room = room  # needed to send messages to the browser

    @function_tool
    async def show_section(self, section: str) -> str:
        """Scroll the guest's web page to one section and highlight it.

        Args:
            section: One of: hours, mains, location.
        """
        if section not in SECTIONS:  # tell the model, so it retries with a real name
            return f"No section {section!r}. Use one of: hours, mains, location."
        message = json.dumps({"action": "show_section", "section": section})
        await self.room.local_participant.publish_data(message, topic="ui")  # index.html listens on "ui"
        return "Done."


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    # The brain: each line is one piece of the pipeline.
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],                                              # is someone talking?
        stt=inference.STT(model="cartesia/ink-whisper", language="zh"),            # Chinese speech -> text
        llm=inference.LLM(model="openai/gpt-4.1-mini"),                            # text -> reply (+ tool calls)
        tts=inference.TTS(model="cartesia/sonic-3", voice=VOICE_ID, language="zh"),  # Chinese reply -> speech
        turn_handling={
            "turn_detection": "vad",                        # have they finished? (ink-whisper can't tell, so use silence)
            "preemptive_generation": {"enabled": False},    # keep off when using tools
        },
    )

    # The face. Start it BEFORE session.start().
    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):  # cold starts occasionally time out, so retry
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    # Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
    faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
    await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
    session.generate_reply(
        instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Putonghua."
    )


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

Step 4. server.py, index.html and FAQs.txt — unchanged

server.py is exactly the same file as in version 3: it still serves the page at / and still hands out tokens at /token.

index.html keeps the same sections (hours, mains, location), the same gold highlight ring and the same listener on the "ui" data channel. Only the one-line introduction now invites visitors to ask in Putonghua:

<p>Ask Lin, our host, in Putonghua about our hours, our main courses or where to find us.</p>

The page itself stays in English, which makes the demo easy to follow: Lin speaks Chinese while the English page scrolls along with her.

FAQs.txt is also untouched — the same four English Q&A pairs, translated by the LLM at call time.

Step 5. Install, set your keys, run

Identical to version 3. In the project folder run uv sync once, then open two terminals:

uv run agent.py dev

and in the second:

uv run server.py

Step 6. Open the page and talk to Lin in Putonghua

Open http://localhost:8080 in your browser. The restaurant page loads with the avatar widget in the bottom-right corner. Click Start and allow the microphone.

Wait for Lin's welcome, then ask in Putonghua, for example:

Ideas to try next

Where to go next