English 繁體中文 简体中文

如何在網頁中使用 LiveKit 和 Synthesia Avatar(版本 5)——廣東話?

林邦源博士 · AI 筆記 · 2026 年 11 月 6 日 · 約 8 分鐘閱讀

在較早的一篇筆記中,餐廳接待員 Lin 搬進了一個 真正的網頁:當她回答問題時,頁面會捲動到她正在講述的部份。在 普通話版本中,她學會了說普通話。

這篇筆記朝著另一個方向作同樣的躍進:Lin 現在會說廣東話 — 是香港人的日常口語,而不是標準書面中文。你問「你哋幾點開門?」, 她會像一位友善的侍應那樣回答,用繁體字,而頁面會捲動到營業時間。她的知識檔案 仍然是英文,網頁也是英文;AI 會即時翻譯。

普通話與廣東話一覽

版本 3(普通話)版本 5(廣東話)
對話語言普通話(國語)香港口語廣東話
耳朵(語音轉文字)Cartesia ink-whisper,設定為中文Speechmatics linden-1,設定為廣東話
聲音(文字轉語音)Cartesia sonic-3,聲音「Jing」Inworld TTS 2,聲音「Xinyi」
知道你何時說完靜音偵測(VAD)由 STT 模型在它的伺服器上判斷
文字簡體繁體,包含 係、唔、喺 等口語字
知識檔案 FAQs.txt英文 — 即時翻譯英文 — 即時翻譯
頁面捲動並標示會(show_section 工具)會(show_section 工具)
新帳戶或金鑰不需要不需要

開始之前你需要甚麼

在自己的電腦上運行

下載 zip 檔案並解壓到一個資料夾。 zip 內有 agent.py、server.py、 index.html、FAQs.txt、pyproject.toml 和 README.md。加入你自己的 .env 檔案,寫入跟之前一樣的五把金鑰。

步驟 1. 設定專案資料夾

六個檔案中有五個 — server.py、FAQs.txt、 pyproject.toml,以及(除了一行之外)index.html — 都跟之前一樣,所以如果你已經有其中一個較早版本的資料夾,可以複製它, 只替換 agent.py。

載有五把秘密金鑰的 .env 檔案也完全一樣。真正改變的只有 agent.py,而所有改動同樣都跟語言有關。

步驟 2. 看看 agent.py — 有甚麼改變

一共有四項改動,而第一項迫出了第二項。

改動 1:新的耳朵,聽得懂廣東話

stt=inference.STT(model="speechmatics/linden-1", language="yue"),

語音轉文字模型 — Lin 的耳朵 — 由 Cartesia 的 ink-whisper 改為 Speechmatics 的 linden-1,並設定 language="yue"(yue 是廣東話的代號)。Cartesia 的模型能處理普通話, 但不能處理廣東話,所以這是一個不同的供應商。

你不需要 Speechmatics 帳戶:模型經 LiveKit Inference 運行, 也就是一直提供之前耳朵和聲音的同一個服務,透過你的 LiveKit 專案計費。

改動 2:輪次偵測回到 STT 模型

turn_handling={
    "turn_detection": "stt",                        # have they finished? (Speechmatics decides, on its servers)
    "preemptive_generation": {"enabled": False},    # keep off when using tools
},

在普通話版本中,我們要用 "vad" — 靠靜音猜測 — 因為 ink-whisper 沒有輪次偵測的訊號。Speechmatics 會回報一輪對話的結束,所以我們可以 切換回 "stt":Lin 現在會等辨識器自己說你講完了,這樣比固定的停頓 更能處理廣東話口語中急促、吞音的結尾。

改動 3:新的廣東話聲音

VOICE_ID = "Xinyi"  # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)
tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"),

聲音由 Cartesia 的普通話 Jing 改為 Inworld 的 Xinyi,並設定 language="yue"。注意 Inworld 的聲音用簡單的名字,不像 Cartesia 那樣 用長長的 ID — 你可以改那一行,試試 Xiaoyin、Jing 或 Yichen(男聲)。avatar 仍然一如以往經 Synthesia 說話。

改動 4:提示詞要求口語廣東話

The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.

這比普通話那一行長,是有原因的。沒有指引的話,會說中文的 LLM 會飄向標準書面中文, 說出來生硬,也不是香港侍應會說的話。點名那六個日常用字,並要求用繁體字, 可以讓 Lin 留在口語廣東話裏。

地址那條規則存在,是因為一個地名可以有好幾種讀法;寫清楚可以確保聲音用香港聽眾 期望的方式說出 九龍塘達之路。後備那一行和歡迎訊息也改了:

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.
session.generate_reply(
    instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
)

其他一切 — show_section 工具、avatar 的重試迴圈、每次通話都 重新讀取 FAQs.txt — 都沒有改變。

步驟 3. 完整的 agent.py

以下是完整的檔案,讓你可以看到四項改動在上下文中的位置:

"""Version 5 (Cantonese): Lin speaks Cantonese. The guest asks in Cantonese and Lin answers in Cantonese.

The pipeline:
    mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser
    VAD: Voice Activity Detection, STT: Speech-to-Text, LLM: Large Language Model, TTS: Text-to-Speech

New in this version: the ears (Speechmatics) and the voice (Inworld) change to models
that handle Cantonese, and the prompt tells Lin to reply in colloquial Hong Kong
Cantonese. Both still run through LiveKit Inference, so no new API key is needed.
FAQs.txt, the prompt and the web page stay in English: the LLM translates on the fly.

Run:  uv run agent.py dev    (terminal 1)
      uv run server.py       (terminal 2, then open http://localhost:8080)
"""

import asyncio
import json
import os
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"  # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25"  # Jenny
VOICE_ID = "Xinyi"  # Inworld Chinese female voice. Others to try: Xiaoyin, Jing, Yichen (male)

FAQS_FILE = Path(__file__).parent / "FAQs.txt"  # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"}       # must match the section ids in index.html

# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.

The guest speaks Cantonese. Always reply in colloquial Hong Kong Cantonese, the way
a friendly waiter talks, written in traditional Chinese characters with spoken
Cantonese words such as 係, 唔, 喺, 嘅, 咗 and 啲. Do not use standard written Chinese
or Putonghua. Reply in Cantonese even though the FAQs below are in English. Say the
address as 九龍塘達之路.

Answer only from these FAQs:

{faqs}

Those are the only things you know. If the guest asks anything the FAQs do not
cover, give the answer to "Any other question?", translated into Cantonese.

The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:

- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")

Do not call show_section for any other question."""


class Host(Agent):
    def __init__(self, room, instructions: str) -> None:
        super().__init__(instructions=instructions)
        self.room = room  # needed to send messages to the browser

    @function_tool
    async def show_section(self, section: str) -> str:
        """Scroll the guest's web page to one section and highlight it.

        Args:
            section: One of: hours, mains, location.
        """
        if section not in SECTIONS:  # tell the model, so it retries with a real name
            return f"No section {section!r}. Use one of: hours, mains, location."
        message = json.dumps({"action": "show_section", "section": section})
        await self.room.local_participant.publish_data(message, topic="ui")  # index.html listens on "ui"
        return "Done."


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    # The brain: each line is one piece of the pipeline.
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],                                              # is someone talking?
        stt=inference.STT(model="speechmatics/linden-1", language="yue"),          # Cantonese speech -> text
        llm=inference.LLM(model="openai/gpt-4.1-mini"),                            # text -> reply (+ tool calls)
        tts=inference.TTS(model="inworld/inworld-tts-2", voice=VOICE_ID, language="yue"),  # Cantonese reply -> speech
        turn_handling={
            "turn_detection": "stt",                        # have they finished? (Speechmatics decides, on its servers)
            "preemptive_generation": {"enabled": False},    # keep off when using tools
        },
    )

    # The face. Start it BEFORE session.start().
    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):  # cold starts occasionally time out, so retry
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    # Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
    faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
    await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
    session.generate_reply(
        instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence of Cantonese."
    )


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

步驟 4. server.py、index.html 和 FAQs.txt

server.py 跟之前完全一樣:它仍然在 / 提供頁面, 仍然在 /token 發出權杖。

index.html 保留同樣的部份(hours、 mains、location)、同樣的金色標示外框,以及同樣監聽 "ui" 數據頻道的監聽器。只有兩行改變:頁面標題和那一行簡介, 現在邀請訪客用廣東話提問:

<p>Ask Lin, our host, in Cantonese about our hours, our main courses or where to find us.</p>

頁面本身仍然是英文,這讓示範容易跟上:Lin 說廣東話,而英文頁面跟着她一起捲動。

FAQs.txt 沒有改動 — 同樣四組英文問答,由 LLM 在通話時翻譯。

步驟 5. 安裝、設定金鑰、運行

與較早版本完全相同。在專案資料夾執行一次 uv sync, 然後開兩個終端機。

終端機 1 — 語音代理 + avatar 引擎:

uv run agent.py dev

終端機 2 — 提供瀏覽器頁面的網頁伺服器:

uv run server.py

步驟 6. 開啟頁面,用廣東話跟 Lin 對話

在瀏覽器開啟 http://localhost:8080。餐廳頁面載入,avatar 小工具在 右下角。點擊 Start 並允許咪高峰權限。

等候 Lin 的歡迎,然後用廣東話提問,例如:

留意那些關鍵字 係、唔、喺、嘅、咗 和 啲:它們告訴你她在說廣東話, 而不是把標準書面中文讀出來。

接下來可以試試

下一步