English 繁體中文 简体中文

如何在网页中使用 LiveKit 和 Synthesia Avatar(版本 5)——粤语?

林邦源博士 · AI 笔记 · 2026 年 11 月 6 日 · 约 8 分钟阅读

在较早的一篇笔记中,餐厅接待员 Lin 搬进了一个 真正的网页:当她回答问题时,页面会滚动到她正在讲述的部分。在 普通话版本中,她学会了说普通话。

这篇笔记朝着另一个方向作同样的跃进:Lin 现在会说粤语 — 是香港人的日常口语,而不是标准书面中文。你问“你哋幾點開門?”, 她会像一位友善的服务员那样回答,用繁体字,而页面会滚动到营业时间。她的知识文件 仍然是英文,网页也是英文;AI 会即时翻译。

普通话与粤语一览

版本 3(普通话)版本 5(粤语)
对话语言普通话(国语)香港口语粤语
耳朵(语音转文字)Cartesia ink-whisper,设置为中文Speechmatics linden-1,设置为粤语
声音(文字转语音)Cartesia sonic-3,声音“Jing”Inworld TTS 2,声音“Xinyi”
知道你何时说完静音检测(VAD)由 STT 模型在它的服务器上判断
文字简体繁体,包含 係、唔、喺 等口语字
知识文件 FAQs.txt英文 — 即时翻译英文 — 即时翻译
页面滚动并标示会(show_section 工具)会(show_section 工具)
新账户或密钥不需要不需要

开始之前你需要什么

在自己的电脑上运行

下载 zip 文件并解压到一个文件夹。 zip 内有 agent.py、server.py、 index.html、FAQs.txt、pyproject.toml 和 README.md。加入你自己的 .env 文件,写入跟之前一样的五个密钥。

步骤 1. 设置项目文件夹

六个文件中有五个 — server.py、FAQs.txt、 pyproject.toml,以及(除了一行之外)index.html — 都跟之前一样,所以如果你已经有其中一个较早版本的文件夹,可以复制它, 只替换 agent.py。

载有五个密钥的 .env 文件也完全一样。真正改变的只有 agent.py,而所有改动同样都跟语言有关。

步骤 2. 看看 agent.py — 有什么改变

一共有四项改动,而第一项逼出了第二项。

改动 1:新的耳朵,听得懂粤语

stt=inference.STT(model="speechmatics/linden-1", language="yue"),

语音转文字模型 — Lin 的耳朵 — 由 Cartesia 的 ink-whisper 改为 Speechmatics 的 linden-1,并设置 language="yue"(yue 是粤语的代码)。Cartesia 的模型能处理普通话, 但不能处理粤语,所以这是一个不同的供应商。

你不需要 Speechmatics 账户:模型经 LiveKit Inference 运行, 也就是一直提供之前耳朵和声音的同一个服务,通过你的 LiveKit 项目计费。

改动 2:轮次检测回到 STT 模型

turn_handling={
    "turn_detection": "stt",                        # have they finished? (Speechmatics decides, on its servers)
    "preemptive_generation": {"enabled": False},    # keep off when using tools
},

在普通话版本中,我们只能退回 "vad" — 靠静音猜测 — 因为 ink-whisper 没有轮次检测信号。Speechmatics 会报告一轮对话的结束,所以我们可以 切换回 "stt":Lin 现在会等识别器自己说你说完了,这样比固定的停顿 更能处理粤语口语中急促、吞音的结尾。

改动 3:新的粤语声音

VOICE_ID = "Xinyi"  # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)
tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"),

声音由 Cartesia 的普通话 Jing 改为 Inworld 的 Xinyi,并设置 language="yue"。注意 Inworld 的声音用简单的名字,不像 Cartesia 那样 用长长的 ID — 你可以改那一行,试试 Xiaoyin、Jing 或 Yichen(男声)。avatar 仍然一如以往经 Synthesia 说话。

改动 4:提示词要求口语粤语

The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.

这比普通话那一行长,是有原因的。没有指引的话,会说中文的 LLM 会飘向标准书面中文, 说出来生硬,也不是香港服务员会说的话。点名那六个日常用字,并要求用繁体字, 可以让 Lin 留在口语粤语里。

地址那条规则存在,是因为一个地名可以有好几种读法;写清楚可以确保声音用香港听众 期望的方式说出 九龍塘達之路。备用那一行和欢迎消息也改了:

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.
session.generate_reply(
    instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
)

其他一切 — show_section 工具、avatar 的重试循环、每次通话都 重新读取 FAQs.txt — 都没有改变。

步骤 3. 完整的 agent.py

以下是完整的文件,让你可以看到四项改动在上下文中的位置:

"""Version 5 (Cantonese): Lin speaks Cantonese. The guest asks in Cantonese and Lin answers in Cantonese.

The pipeline:
    mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
    VAD: Voice Activity Detection, STT: Speech-to-Text, LLM: Large Language Model, TTS: Text-to-Speech

New in this version: the ears (Speechmatics) and the voice (Inworld) change to models
that handle Cantonese, and the prompt tells Lin to reply in colloquial Hong Kong
Cantonese. Both still run through LiveKit Inference, so no new API key is needed.
FAQs.txt, the prompt and the web page stay in English: the LLM translates on the fly.

Run:  uv run agent.py dev    (terminal 1)
      uv run server.py       (terminal 2, then open http://localhost:8080)
"""

import asyncio
import json
import os
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"  # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25"  # Jenny
VOICE_ID = "Xinyi"  # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)

FAQS_FILE = Path(__file__).parent / "FAQs.txt"  # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"}       # must match the section ids in index.html

# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.

The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.

Answer only from these FAQs:

{faqs}

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.

The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:

- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")

Do not call show_section for any other question."""


class Host(Agent):
    def __init__(self, room, instructions: str) -> None:
        super().__init__(instructions=instructions)
        self.room = room  # needed to send messages to the browser

    @function_tool
    async def show_section(self, section: str) -> str:
        """Scroll the guest's web page to one section and highlight it.

        Args:
            section: One of: hours, mains, location.
        """
        if section not in SECTIONS:  # tell the model, so it retries with a real name
            return f"No section {section!r}. Use one of: hours, mains, location."
        message = json.dumps({"action": "show_section", "section": section})
        await self.room.local_participant.publish_data(message, topic="ui")  # index.html listens on "ui"
        return "Done."


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    # The brain: each line is one piece of the pipeline.
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],                                              # is someone talking?
        stt=inference.STT(model="speechmatics/linden-1", language="yue"),          # Cantonese speech -> text
        llm=inference.LLM(model="openai/gpt-4.1-mini"),                            # text -> reply (+ tool calls)
        tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"),  # Cantonese reply -> speech
        turn_handling={
            "turn_detection": "stt",                        # have they finished? (Speechmatics decides, on its servers)
            "preemptive_generation": {"enabled": False},    # keep off when using tools
        },
    )

    # The face. Start it BEFORE session.start().
    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):  # cold starts occasionally time out, so retry
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    # Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
    faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
    await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
    session.generate_reply(
        instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
    )


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

步骤 4. server.py、index.html 和 FAQs.txt

server.py 跟之前完全一样:它仍然在 / 提供页面, 仍然在 /token 发放令牌。

index.html 保留同样的部分(hours、 mains、location)、同样的金色标示外框,以及同样监听 "ui" 数据频道的监听器。只有两行改变:页面标题和那一行简介, 现在邀请访客用粤语提问:

<p>Ask Lin, our host, in Cantonese about our hours, our main courses or where to find us.</p>

页面本身仍然是英文,这让演示容易跟上:Lin 说粤语,而英文页面跟着她一起滚动。

FAQs.txt 没有改动 — 同样四组英文问答,由 LLM 在通话时翻译。

步骤 5. 安装、设置密钥、运行

与较早版本完全相同。在项目文件夹运行一次 uv sync, 然后打开两个终端。

终端 1 — 语音代理 + avatar 引擎:

uv run agent.py dev

终端 2 — 提供浏览器页面的网页服务器:

uv run server.py

步骤 6. 打开页面,用粤语跟 Lin 对话

在浏览器打开 http://localhost:8080。餐厅页面载入,avatar 小部件在 右下角。点击 Start 并允许麦克风权限。

等候 Lin 的欢迎,然后用粤语提问,例如:

留意那些关键词 係、唔、喺、嘅、咗 和 啲:它们告诉你她在说粤语, 而不是把标准书面中文读出来。

接下来可以试试

下一步