English 繁體中文 简体中文

如何在網頁中使用 LiveKit 和 Synthesia Avatar 並在網頁點餐(版本 6)——英文?

林邦源博士 · AI 筆記 · 2026 年 11 月 6 日 · 約 9 分鐘閱讀

在較早的一篇筆記中,餐廳接待員 Lin 搬進了一個 真正的網頁。她當時只有一個工具 show_section,也只有一個任務:回答一條問題, 然後把頁面捲動到她正在講述的那個部份。

本篇筆記給她一份真正的工作。頁面不再是三個部份加一段影片;它是一整個餐廳網站, 有菜單、酒單、價格、致敏原,還有一張帳單。Lin 變成總侍應, 她逐道菜為你點餐:前菜、頭盤、主菜、甜品,然後是酒。她仍然會說話; 但現在她一邊說話,一邊在頁面上行動 — 把你不能吃的菜式變灰、 打開她正在介紹的那道菜的卡片,並把你選的東西加到角落那張持續更新的帳單上。

回答問題與接受點餐:一覽對照

版本 3(回答問題)版本 6(接受點餐)
頁面一頁、三個部份,數位人在角落完整的餐廳網站:菜單、酒、用餐空間,以及一張持續更新的帳單
Lin 可以呼叫的工具1 個 — show_section4 個 — show_section、filter_menu、focus_dish、add_to_order
她所知道的FAQs.txt 裏 4 組問答13 組 — 整個菜單、價格、飲食要求、致敏原和酒
對話回答,然後停止有引導:一次一道菜、兩個選項,然後是酒
頁面的反應捲動並標示捲動、把菜式變灰、打開菜式卡片、加入帳單
折扣沒有每道菜各點一款時,食物有九折
耳朵Cartesia ink-2相同,另加一份要留意的菜單詞彙
聲音Cartesia sonic-3.6,聲音 Skylar相同
網頁檔案index.html,全部內容都在裏面index.html(細小)+ site.js + site.css

開始之前你需要甚麼

在自己的電腦上運行

下載 zip 檔案並 解壓到一個資料夾。zip 內有 agent.py、server.py、 index.html、site.js、site.css、FAQs.txt、 pyproject.toml 和 README.md。加入你自己的 .env 檔案, 用與之前相同的五條金鑰。

步驟 1. 設定專案資料夾

如果你已經有其中一個較早的資料夾,保留 server.py、 pyproject.toml 和 .env,取代 agent.py、 index.html 和 FAQs.txt,並加入兩個新檔案 site.js 和 site.css。在 server.py 中,你還需要兩條 額外的路由,因為頁面現在會要求這兩個檔案(步驟 4 會展示)。

.env 檔案完全沒有改變:同樣的 LiveKit 金鑰、Synthesia 金鑰和 數位人 ID。

步驟 2. 看看 agent.py — 有甚麼改變

四項改變:三個工具是新的、提示詞變成總侍應的劇本、耳朵得到一份菜單詞彙, 還有一個小小的輔助函數負責發送所有工具訊息。

改變 1:一個輔助函數,四個工具

async def _ui(self, **message) -> str:
    """Send a tiny message to the browser. step09/site.js does the rest."""
    await self.room.local_participant.publish_data(json.dumps(message), topic="ui")
    return "Done."

@function_tool
async def show_section(self, section: str) -> str:
    """Scroll the page to a part of the menu and highlight it.

    Args:
        section: One of: meal, openers, first, mains, desserts, wine, room.
    """
    return await self._ui(action="show_section", section=section)

@function_tool
async def filter_menu(self, diet: str) -> str:
    """Grey out every dish on the page the guest cannot eat.

    Args:
        diet: One of: none, vegetarian, vegan, pescatarian, nut_free.
    """
    return await self._ui(action="filter_menu", diet=diet)

@function_tool
async def focus_dish(self, dish: str) -> str:
    """Open the detail card for one dish: kitchen note and allergens.

    Args:
        dish: A dish ID from the menu, for example salmon or tofu.
    """
    return await self._ui(action="focus_dish", dish=dish)

版本 3 在它唯一的工具裏手工組裝 JSON。有了四個工具,那就會是同樣兩行複製四次, 所以一個小小的私有輔助函數 _ui 只做一次:它把你傳入的任何東西變成 JSON,並在 "ui" 數據頻道上發佈。之後每個工具都只有一行, 而它的 docstring — AI 模型真正會讀的那一段 — 說明這個工具接受甚麼。

留意這裏不再有的東西:SECTIONS 檢查。版本 3 會拒絕未知的部份名稱。 現在頁面本身會忽略它不認識的名稱,所以代理可以保持簡單;唯一仍然做驗證的地方 是點餐,因為一道不存在的菜式絕不應該出現在帳單上。

改變 2:接受點餐

@function_tool
async def add_to_order(self, dish: str) -> str:
    """Add a dish or a glass of wine to the guest's order, shown as a running tab on the page.

    Args:
        dish: A dish ID from the menu, or a wine ID: red, white or alcoholfree.
    """
    # Accept "red", "Red Wine" or "alcohol-free" alike, the same way site.js does.
    dish = re.sub(r"[^a-z]", "", dish.lower()).removeprefix("the").replace("wine", "")
    if dish not in ORDER_IDS:  # tell the model, so it retries with a real ID
        return f"Not added: {dish!r} is not an ID. Use one of: {', '.join(sorted(ORDER_IDS))}."
    await self._ui(action="add_to_order", dish=dish)
    # Remind the model to move the page along with the conversation.
    section = next(c for c, dishes in COURSES.items() if dish in dishes)
    if section in NEXT_SECTION:
        nxt = NEXT_SECTION[section]
        return f"Added {dish} to the order. When you introduce the next part, call show_section(\"{nxt}\") too."
    return f"Added {dish} to the order."

有兩個想法值得抄進你自己的代理。第一,清理那一行:模型可能送出 "Red Wine"、"red" 或 "alcohol-free", 而客人可能會說「the red」(那杯紅的);把所有不是字母的字元剝掉,再移除 「the」和「wine」,就會把它們全部變成頁面使用的那一個 ID。 第二,回覆是一個提示。工具結果不只是一張收據:透過回傳 「當你介紹下一部份時,也呼叫 show_section("mains")」, 代理會悄悄地引導自己,讓頁面與對話保持同步。

菜式清單本身放在類別之上,緊鄰它們所屬的提示詞:

COURSES = {
    "openers": ["oysters", "bread", "salad"],
    "first": ["soup", "prawns", "dumplings"],
    "mains": ["salmon", "beef", "chicken", "pork", "tofu"],
    "desserts": ["cheesecake", "chocolate", "sorbet"],
    "wine": ["red", "white", "alcoholfree"],
}
ORDER_IDS = {dish for dishes in COURSES.values() for dish in dishes}
NEXT_SECTION = dict(zip(COURSES, list(COURSES)[1:]))  # openers -> first -> mains -> desserts -> wine

COURSES 必須與 site.js 中的 MENU 和 WINES 一致。如果你在那裏加一道菜,這裡也要加,否則 Lin 會禮貌地 拒絕為你點它。

改變 3:提示詞變成劇本

版本 3 的提示詞是一本規則手冊:從 FAQs 回答,然後就那條問題捲動到這個部份。 版本 6 的提示詞是一份職位描述。事實搬了出去 — 價格、 飲食要求和致敏原現在都在 FAQs.txt 裏 — 而提示詞只說明 該怎樣行事,以及工具接受哪些 ID。正是這一部分令示範感覺像一間餐廳, 而不是一個聊天機械人:

Work like a real head waiter. Take the guest through the courses in order:
openers, then first, then mains, then desserts, asking about one course at a time.
Offer two options rather than ten. If they skip a course, let them, but mention
once that one dish from each course gets 10% off. After desserts, offer a glass of
wine once, and let the guest pick any one; if they say no, that is fine. Do not
suggest a wine for a particular dish.

小規則,大效果。「兩個選項而不是十個」令口頭回答保持簡短。「一次一道菜」 給對話一個形狀,也令 show_section 在每一輪都有事可做。 「不要為某一道菜建議配酒」阻止模型發明它不可能知道的配搭。

還有一段關於聽錯的字詞的新文字,而它是整個檔案中最實用的一段:

What the guest says reaches you through speech-to-text, which sometimes mishears.
If their words make no sense, they are probably not a question: think what they
most likely said, given what you just asked. For example, right after you offer
wine, "wet why" is probably "red wine" or "white wine". Check with them in a few
words, such as "Red wine, is that right?", or ask them to say it again. Do not use
the "Any other question?" answer for words you did not understand.

沒有它,一個聽錯的菜名會把 Lin 送去後備答案,聽起來像一個放棄了的機械人。 有了它,她會改為問一條簡短的問題 — 這正是一位侍應會做的事。

改變 4:耳朵得到一份詞彙表

# Words the speech-to-text should listen out for, so "red wine" is not heard as "wet why".
KEYTERMS = [
    "YouFantasy", "Oysters", "Garlic Bread", "Green Salad", "Tomato Soup", "Grilled Prawns",
    "Vegetable Dumplings", "Salmon", "Beef", "Roast Chicken", "Pork Belly", "Crispy Tofu",
    "Cheesecake", "Chocolate Cake", "Mango Sorbet", "Red Wine", "White Wine", "Alcohol-Free",
    "openers", "first course", "mains", "desserts", "vegetarian", "vegan", "nut free",
]
stt=inference.STT(model="cartesia/ink-2", extra_kwargs={"keyterm": KEYTERMS}),  # speech -> text

這些字以 keyterms(關鍵詞)的形式傳給語音辨識器:是提示, 不是規則。餐廳名稱、菜名和飲食詞彙,正是一般模型最容易弄錯的幾類; 而弄錯菜名比弄錯形容詞更糟 — 它會上到帳單。 流程的其餘部分與版本 3 相同:openai/gpt-4.1-mini 做腦袋, cartesia/sonic-3.6 配 Skylar 聲音做嗓子,turn detection 設在 "stt",以及同一個 Synthesia avatar,連同它的三次冷啟動重試。

步驟 3. 完整的 agent.py

以下是完整檔案,讓你可以看到四項改變在上下文中的樣子:

"""Version 6 (English): the full restaurant. Lin is a head waiter who takes the guest's whole order.

The pipeline:
    mic -> VAD -> STT -> turn detection -> LLM (+ tools) -> TTS -> Synthesia avatar -> browser

New in this version: the page is a whole restaurant website (site.js + site.css) and Lin has
four tools instead of one - show_section, filter_menu, focus_dish and add_to_order - so she
can grey out dishes for an allergy, open a dish card and keep a running tab. The prompt turns
her into a head waiter who guides the guest through four courses, and the speech-to-text gets
a list of menu words so "red wine" is not heard as "wet why".

Run:  uv run agent.py dev    (terminal 1)
      uv run server.py       (terminal 2, then open http://localhost:8080)
"""

import asyncio
import json
import os
import re
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"  # must match server.py
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25"  # Jenny
VOICE_ID = "db6b0ed5-d5d3-463d-ae85-518a07d3c2b4"  # Cartesia "Skylar"

# Everything add_to_order accepts, by page section, in the order Lin takes the guest
# through them. Must match MENU and WINES in step09/site.js.
COURSES = {
    "openers": ["oysters", "bread", "salad"],
    "first": ["soup", "prawns", "dumplings"],
    "mains": ["salmon", "beef", "chicken", "pork", "tofu"],
    "desserts": ["cheesecake", "chocolate", "sorbet"],
    "wine": ["red", "white", "alcoholfree"],
}
ORDER_IDS = {dish for dishes in COURSES.values() for dish in dishes}
NEXT_SECTION = dict(zip(COURSES, list(COURSES)[1:]))  # openers -> first -> mains -> desserts -> wine

# Words the speech-to-text should listen out for, so "red wine" is not heard as "wet why".
KEYTERMS = [
    "YouFantasy", "Oysters", "Garlic Bread", "Green Salad", "Tomato Soup", "Grilled Prawns",
    "Vegetable Dumplings", "Salmon", "Beef", "Roast Chicken", "Pork Belly", "Crispy Tofu",
    "Cheesecake", "Chocolate Cake", "Mango Sorbet", "Red Wine", "White Wine", "Alcohol-Free",
    "openers", "first course", "mains", "desserts", "vegetarian", "vegan", "nut free",
]

FAQS_FILE = Path(__file__).parent / "FAQs.txt"  # the avatar's knowledge

# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
# The facts (prices, diets, allergens, wine) live in FAQs.txt; this prompt only says
# how to behave and which IDs the tools take.
INSTRUCTIONS = """\
You are Lin, the head waiter at YouFantasy, a made-up restaurant. You are a
real-time avatar in a spoken conversation, so answer in one to three short
sentences and never use markdown, lists, or URLs.

Answer only from these FAQs:

{faqs}

Those are the only facts you know. If the guest asks for information the FAQs do
not cover, use the answer to "Any other question?" word for word. Taking their
order and guiding them through the courses is your job, not a question, so carry
on with that normally.

What the guest says reaches you through speech-to-text, which sometimes mishears.
If their words make no sense, they are probably not a question: think what they
most likely said, given what you just asked. For example, right after you offer
wine, "wet why" is probably "red wine" or "white wine". Check with them in a few
words, such as "Red wine, is that right?", or ask them to say it again. Do not use
the "Any other question?" answer for words you did not understand.

Your tools take these IDs. Say the dish names out loud, never the IDs, and never
read prices as digits:

  openers   oysters, bread (Garlic Bread), salad (Green Salad)
  first     soup (Tomato Soup), prawns (Grilled Prawns), dumplings (Vegetable Dumplings)
  mains     salmon, beef, chicken (Roast Chicken), pork (Pork Belly), tofu (Crispy Tofu)
  desserts  cheesecake, chocolate (Chocolate Cake), sorbet (Mango Sorbet)
  wine      red (Red Wine), white (White Wine), alcoholfree (Alcohol-Free)

Use your tools while you talk, not instead of talking:

- show_section when you move the guest to a new part of the page: meal for how
  the menu works, a course name for that course, wine for wine, room for the
  hours or the address
- filter_menu the moment they mention a dietary need, and again if they change it
- focus_dish when you describe one dish in any detail
- add_to_order only once they have actually chosen something

Every time you start talking about a course or the wine, call show_section for it
in the same reply, so the page always shows what you are talking about. This
matters most right after add_to_order: when you add a dish and move on to the next
course, call both tools, for example add_to_order("soup") and show_section("mains").

Work like a real head waiter. Take the guest through the courses in order:
openers, then first, then mains, then desserts, asking about one course at a time.
Offer two options rather than ten. If they skip a course, let them, but mention
once that one dish from each course gets 10% off. After desserts, offer a glass of
wine once, and let the guest pick any one; if they say no, that is fine. Do not
suggest a wine for a particular dish.

The guest can ask about or order wine at any time, not only after desserts. If
they say "red wine", "white wine" or "alcohol-free", answer from the FAQs, and if
they want it, add it with add_to_order, then carry on where you were."""


class HeadWaiter(Agent):
    def __init__(self, room, instructions: str) -> None:
        super().__init__(instructions=instructions)
        self.room = room

    async def _ui(self, **message) -> str:
        """Send a tiny message to the browser. step09/site.js does the rest."""
        await self.room.local_participant.publish_data(json.dumps(message), topic="ui")
        return "Done."

    @function_tool
    async def show_section(self, section: str) -> str:
        """Scroll the page to a part of the menu and highlight it.

        Args:
            section: One of: meal, openers, first, mains, desserts, wine, room.
        """
        return await self._ui(action="show_section", section=section)

    @function_tool
    async def filter_menu(self, diet: str) -> str:
        """Grey out every dish on the page the guest cannot eat.

        Args:
            diet: One of: none, vegetarian, vegan, pescatarian, nut_free.
        """
        return await self._ui(action="filter_menu", diet=diet)

    @function_tool
    async def focus_dish(self, dish: str) -> str:
        """Open the detail card for one dish: kitchen note and allergens.

        Args:
            dish: A dish ID from the menu, for example salmon or tofu.
        """
        return await self._ui(action="focus_dish", dish=dish)

    @function_tool
    async def add_to_order(self, dish: str) -> str:
        """Add a dish or a glass of wine to the guest's order, shown as a running tab on the page.

        Args:
            dish: A dish ID from the menu, or a wine ID: red, white or alcoholfree.
        """
        # Accept "red", "Red Wine" or "alcohol-free" alike, the same way site.js does.
        dish = re.sub(r"[^a-z]", "", dish.lower()).removeprefix("the").replace("wine", "")
        if dish not in ORDER_IDS:  # tell the model, so it retries with a real ID
            return f"Not added: {dish!r} is not an ID. Use one of: {', '.join(sorted(ORDER_IDS))}."
        await self._ui(action="add_to_order", dish=dish)
        # Remind the model to move the page along with the conversation.
        section = next(c for c, dishes in COURSES.items() if dish in dishes)
        if section in NEXT_SECTION:
            nxt = NEXT_SECTION[section]
            return f"Added {dish} to the order. When you introduce the next part, call show_section(\"{nxt}\") too."
        return f"Added {dish} to the order."


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    # The brain: each line is one piece of the pipeline.
    session = AgentSession(
        vad=ctx.proc.userdata["vad"],                                   # is someone talking?
        stt=inference.STT(model="cartesia/ink-2", extra_kwargs={"keyterm": KEYTERMS}),  # speech -> text
        llm=inference.LLM(model="openai/gpt-4.1-mini"),                 # text -> reply (swap in any LLM)
        tts=inference.TTS(model="cartesia/sonic-3.6", voice=VOICE_ID),  # reply -> speech
        turn_handling={
            "turn_detection": "stt",                        # have they finished their thought?
            "preemptive_generation": {"enabled": False},    # keep off when using tools
        },
    )

    # The face: this is the whole Synthesia integration. Start it BEFORE session.start().
    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):  # cold starts occasionally time out, so retry
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    # Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
    faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
    await session.start(agent=HeadWaiter(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
    session.generate_reply(instructions="Welcome the guest to YouFantasy in one short sentence.")


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

步驟 4. server.py、index.html、site.js 和 FAQs.txt

server.py 仍然在 / 提供頁面,也仍然在 /token 發出權杖。它多了一條路由,因為 index.html 現在 會向瀏覽器要求兩個額外檔案:

async def asset(request: web.Request) -> web.FileResponse:
    # /site.js and /site.css, next to index.html. Only .js and .css, so agent.py and FAQs stay private.
    path = Path(__file__).parent / request.match_info["file"]
    if not path.is_file():
        raise web.HTTPNotFound()
    return web.FileResponse(path, headers={"Cache-Control": "no-store"})


app = web.Application()
app.router.add_get("/", index)
app.router.add_get("/token", token)
app.router.add_get(r"/{file:[\w-]+\.(?:js|css)}", asset)

把路由限制在 .js 和 .css 是刻意的:這意味着 agent.py、FAQs.txt 和你的 .env 不能被訪客下載。

index.html 現在很細小 — 大約八十行。它只装着數位人影片、 Start 按鈕和 LiveKit 連線,別無其他。整間餐廳由 site.js 建立, 而 site.js 對數位人、LiveKit 或 Synthesia 一無所知。它只是 發佈一個全域物件,每個工具對應一個方法:

const SITE = {
  show_section({ section }) { ... },   // scroll there and draw a ring around it
  filter_menu({ diet }) { ... },       // grey out every dish the guest cannot eat
  focus_dish({ dish }) { ... },        // open the card: kitchen note, allergens
  add_to_order({ dish }) { ... },      // add it to the running tab
};

兩者之間的橋樑是 index.html 裏的四行。每次工具呼叫都會經 "ui" 數據頻道到達,而動作名稱會揀選方法:

room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
  if (topic !== 'ui') return;
  const { action, ...args } = JSON.parse(new TextDecoder().decode(payload));
  SITE[action]?.(args);
});

這個分工是本篇筆記中最可重用的想法。因為 site.js 不知道數位人的存在, 同一個頁面在代理關掉時仍然可用:你可以手動點擊一道菜、打開它的卡片、 把它加入訂單,並看着九折出現。而因為代理不知道頁面是怎樣建成的, 你可以重新設計或重建網站,而無需觸碰 agent.py。

site.js 也負責兩件與 AI 無關的餐廳邏輯:飲食篩選 (none、vegetarian、vegan、 pescatarian、nut_free)和帳單 — 食物按菜別分組,酒保持分開且永不打折,而當四道菜齊備時, 食物有九折。

const food = order.filter(courseOf);
const fullMeal = COURSES.every((c) => food.some((d) => courseOf(d) === c));
const discount = fullMeal ? food.reduce((t, d) => t + d.price, 0) * DISCOUNT : 0;

FAQs.txt 由四條問題增至十三條:附價格的菜單、四道菜、杯裝酒, 以及真實客人會問的問題 — 哪些菜式是純素、素食、海鮮,或含有果仁。 Lin 每次通話都會重新讀取它,所以你可以改一個價格,並在下一次對話聽到 新的價格。

步驟 5. 安裝、設定金鑰、運行

與之前一樣。在專案資料夾執行一次 uv sync,然後開兩個終端機。

終端機 1 — 語音代理 + 數位人引擎:

uv run agent.py dev

終端機 2 — 提供瀏覽器頁面的網頁伺服器:

uv run server.py

步驟 6. 開啟頁面,點一頓飯

在瀏覽器開啟 http://localhost:8080。餐廳網站載入,數位人小工具在 右下角。點擊 Start 並允許咪高峰權限。等候 Lin 的歡迎, 然後試一頓完整的飯:

完成四道菜,帳單會顯示 全餐,食物九折 和新的總額。 要求一道不在菜單上的菜,她不會憑空發明:點餐工具會拒絕, 而提示詞叫她說出菜名,永不說 ID,也永不把價格逐個數字讀出。

接下來可以試試

下一步