English 繁體中文 简体中文

如何在网页中使用 LiveKit 和 Synthesia Avatar(版本 4)——普通话?

林邦源博士 · AI 笔记 · 2026 年 11 月 6 日 · 约 8 分钟阅读

在版本 2(英文)中,餐厅接待员 Lin 用一个纯文本文件里的知识, 回答你口头提出的问题。在版本 3中,她搬进了一个 真正的网页:当她回答时,页面会滚动到她正在讲述的部分。

版本 4 再做一个改动:Lin 现在说普通话。你用普通话提问,她用普通话的声音回答, 而页面仍然会随着她说话而滚动。有趣的地方是:她的知识文件仍然是英文,网页也是英文 — AI 模型即时把一切翻译过来。

版本 3 与版本 4 一览

版本 3版本 4
对话语言英文普通话
耳朵(语音转文字)Cartesia ink-2Cartesia ink-whisper,设置为中文
声音(文字转语音)Cartesia sonic-3.6,声音“Skylar”Cartesia sonic-3,声音“Jing”(普通话)
判断你何时说完由 STT 模型告知静音检测(VAD)
知识文件 FAQs.txt英文英文 — 即时翻译
网页文字英文英文(不变)
页面滚动并高亮是(show_section 工具)是(show_section 工具)

开始之前你需要什么

在自己的电脑上运行

下载 zip 文件并解压到 一个文件夹。zip 里有 agent.py、server.py、index.html、 FAQs.txt、pyproject.toml 和 README.md。加入你自己的 .env 文件,使用与版本 3 相同的五个密钥。

步骤 1:设置项目文件夹

六个文件中有四个 — server.py、index.html、FAQs.txt 和 pyproject.toml — 与版本 3 完全相同,所以如果你已经有那个文件夹, 可以复制它,只替换 agent.py。

装着你五个密钥的 .env 文件与版本 3 完全一样。pyproject.toml 里的包也没有变化 — 说中文的模型来自 LiveKit 已经连接的同一个 Cartesia 服务。

步骤 2:看看 agent.py — 有什么变化

这个文件与版本 3 几乎相同。只有四处改动,而且全部与语言有关。

改动 1:能听懂中文的新耳朵

stt=inference.STT(model="cartesia/ink-whisper", language="zh"),

语音转文字模型 — Lin 的耳朵 — 从 cartesia/ink-2 改为 cartesia/ink-whisper,并加上 language="zh" (zh 是中文的代码)。这个模型听普通话,并用中文字写下你说的话。

这个切换是被迫的:ink-2 只懂英文,所以不能保留来处理普通话。 ink-whisper 才是能够处理的多语言模型。

改动 2:新的普通话声音

VOICE_ID = "6eb8965c-e295-47bd-a9e4-3eeebb3abcff"  # Cartesia "Jing", a Mandarin female voice
tts=inference.TTS(model="cartesia/sonic-3", voice=VOICE_ID, language="zh"),

文字转语音的声音改为 Jing,一个普通话女声,同样加上 language="zh"。数字人仍然像以前一样通过 Synthesia 说话 — 只有她的声音是新的。

改动 3:用另一种方式判断你已经说完

turn_handling={
    "turn_detection": "vad",                        # have they finished? (ink-whisper can't tell, so use silence)
    "preemptive_generation": {"enabled": False},    # keep off when using tools
},

在版本 3 中,STT 模型本身会报告你何时说完一句话。新的 ink-whisper 模型做不到 — 与 ink-2 不同,它没有轮次检测的信号 — 所以我们把 turn_detection 改为 "vad":程序现在只靠察觉到一段沉默, 就判断你已经说完。

改动 4:提示词多了两行

The guest speaks Putonghua (Mandarin Chinese). Always reply in Putonghua, written in
simplified Chinese characters, even though the FAQs below are in English.

这就是版本 4 的核心。FAQs.txt 里的常见问题仍然是英文,所以你仍然可以 像以前一样编辑它们。LLM 读取英文答案,然后用普通话说出来。兜底的那句话也略有改变:

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Putonghua.

而欢迎语现在也要求用普通话:

session.generate_reply(
    instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Putonghua."
)

其他一切 — show_section 工具、数字人的重试循环、每次通话都重新读取 FAQs.txt — 都与版本 3 相同。

步骤 3:完整的 agent.py

以下是完整文件,让你看清这四处改动的上下文:

"""Version 4: Lin speaks Putonghua. The guest asks in Putonghua and Lin answers in Putonghua.

The pipeline:
    mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
    VAD: Voice Activity Detection, STT: Speech-to-Text, LLM: Large Language Model, TTS: Text-to-Speech

New in version 4: the ears and the voice change to models that handle Chinese, and
the prompt tells Lin to reply in Putonghua. FAQs.txt, the prompt and the web page
stay in English: the LLM translates on the fly.

Run:  uv run agent.py dev    (terminal 1)
      uv run server.py       (terminal 2, then open http://localhost:8080)
"""

import asyncio
import json
import os
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"  # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25"  # Jenny
VOICE_ID = "6eb8965c-e295-47bd-a9e4-3eeebb3abcff"  # Cartesia "Jing", a Mandarin female voice

FAQS_FILE = Path(__file__).parent / "FAQs.txt"  # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"}       # must match the section ids in index.html

# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.

The guest speaks Putonghua (Mandarin Chinese). Always reply in Putonghua, written in
simplified Chinese characters, even though the FAQs below are in English.

Answer only from these FAQs:

{faqs}

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Putonghua.

The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:

- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")

Do not call show_section for any other question."""


class Host(Agent):
    def __init__(self, room, instructions: str) -> None:
        super().__init__(instructions=instructions)
        self.room = room  # needed to send messages to the browser

    @function_tool
    async def show_section(self, section: str) -> str:
        """Scroll the guest's web page to one section and highlight it.

        Args:
            section: One of: hours, mains, location.
        """
        if section not in SECTIONS:  # tell the model, so it retries with a real name
            return f"No section {section!r}. Use one of: hours, mains, location."
        message = json.dumps({"action": "show_section", "section": section})
        await self.room.local_participant.publish_data(message, topic="ui")  # index.html listens on "ui"
        return "Done."


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    # The brain: each line is one piece of the pipeline.
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],                                              # is someone talking?
        stt=inference.STT(model="cartesia/ink-whisper", language="zh"),            # Chinese speech -> text
        llm=inference.LLM(model="openai/gpt-4.1-mini"),                            # text -> reply (+ tool calls)
        tts=inference.TTS(model="cartesia/sonic-3", voice=VOICE_ID, language="zh"),  # Chinese reply -> speech
        turn_handling={
            "turn_detection": "vad",                        # have they finished? (ink-whisper can't tell, so use silence)
            "preemptive_generation": {"enabled": False},    # keep off when using tools
        },
    )

    # The face. Start it BEFORE session.start().
    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):  # cold starts occasionally time out, so retry
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    # Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
    faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
    await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
    session.generate_reply(
        instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Putonghua."
    )


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

步骤 4:server.py、index.html 和 FAQs.txt — 没有改动

server.py 与版本 3 是同一个文件:它仍然在 / 提供页面, 并且仍然在 /token 发放令牌。

index.html 保留相同的部分(hours、mains、 location)、相同的金色高亮外框,以及相同的 "ui" 数据通道监听器。 只有那一行介绍文字,现在邀请访客用普通话提问:

<p>Ask Lin, our host, in Putonghua about our hours, our main courses or where to find us.</p>

页面本身仍然是英文,这让演示更容易跟上:Lin 说中文,而英文页面随着她滚动。

FAQs.txt 同样没有被改动 — 同样四组英文问答,由 LLM 在通话时翻译。

步骤 5:安装、设置你的密钥、运行

与版本 3 完全相同。在项目文件夹中运行一次 uv sync,然后打开两个终端:

uv run agent.py dev

然后在第二个终端:

uv run server.py

步骤 6:打开页面,用普通话跟 Lin 说话

在浏览器打开 http://localhost:8080。餐厅页面会加载,数字人小组件在右下角。 点击 Start 并允许麦克风权限。

等 Lin 的欢迎语,然后用普通话提问,例如:

接下来可以试试

下一步看什么