How to use LiveKit and Synthesia Avatar with webpage (version 3)?
In version 1 the avatar's knowledge was hardcoded in Python, and
in version 2 it moved to a plain text file called
FAQs.txt. But in both versions the browser page was just a video and a Start
button — the avatar could talk, yet it could not do anything on the page.
Version 3 puts the avatar inside a real web page. Lin, the host of the made-up restaurant
YouFantasy, sits in a small widget in the bottom-right corner of the restaurant's site. When a
guest asks "When are you open?", she answers out loud and scrolls the page to the
opening-hours section, drawing a gold ring around it. She does this through her first
tool: a function called show_section.
The pipeline
browser mic → VAD → STT → turn detection → LLM (+ tools) → TTS → Synthesia avatar → browser video
The same five pieces as version 2, wired together over LiveKit's real-time transport, plus one new ability: the LLM can call tools. A tool is a small Python function the model is allowed to run. Here the tool sends a tiny message from the agent to the browser, and the page reacts — all in a fraction of a second, while Lin keeps talking.
What you need
- A Mac (the same steps work on Windows and Linux too).
- Homebrew installed and the
uvpackage manager — if you do not haveuv, runbrew install uv. - A LiveKit Cloud account (the free Build plan is enough).
- A Synthesia account. You need at least one avatar created there (the free plan lets you try).
- Your LiveKit and Synthesia credentials, which you copy into a file called
.env— see below.
Set up LiveKit Cloud
Sign up at cloud.livekit.io. In the dashboard, click Projects → Create Project, then open the project and go to Keys. Copy the three values:
LIVEKIT_URL— looks likewss://your-project.livekit.cloud.LIVEKIT_API_KEY— a short string likeAPIxxxxxxxxxxxxx.LIVEKIT_API_SECRET— a longer string, shown only once.
Set up Synthesia
Sign in at app.synthesia.io.
Go to Avatars and pick or create an avatar — each one has a UUID, shown in the
URL when you open it (it looks like 8788bef1-8020-46e0-a8f4-510ea9989b25).
Copy that as SYNTHESIA_AVATAR_ID.
Then go to API Keys
and create a key — that is SYNTHESIA_API_KEY.
Create your .env
In your project folder, create a file named .env with these five lines:
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=APIxxxxxxxxxxxxx
LIVEKIT_API_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SYNTHESIA_API_KEY=syn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SYNTHESIA_AVATAR_ID=8788bef1-8020-46e0-a8f4-510ea9989b25
Important: never put .env on GitHub or any public site — it contains your live credentials.
Running it on your own computer
Download the zip file and
unzip it to a folder. The zip holds agent.py, server.py,
index.html, FAQs.txt, pyproject.toml, and README.md.
Add your own .env file with the keys, as shown above. Then open the folder in
Visual Studio Code and run these two commands in two separate Terminal windows:
Terminal 1 — the voice agent + avatar engine:
uv sync
uv run agent.py dev
Terminal 2 — the web server that serves the browser page:
uv run server.py
uv sync only needs to run the first time — it installs Python 3.13 and every
package. After that you just start the agent in one terminal and the server in the other.
Open the page in your browser
With both terminals running, open http://127.0.0.1:8080 in Chrome, Safari, or Edge. You will see the YouFantasy restaurant page, with Lin's widget in the bottom-right corner. Click Start, allow microphone permission, and wait a few seconds while the agent and avatar join the room. Then ask a question, for example "Where are you located?" or "What main courses do you serve?" — and watch the page: as Lin answers, it scrolls to the matching section and draws a gold ring around it.
New in version 3: the show_section tool
How does Lin move the page? Three short pieces work together.
1. The prompt tells her when. Inside agent.py, Lin's instructions
now end with a small rule book: when you answer about opening hours, also call
show_section("hours"); for main courses, call show_section("mains");
for the address, call show_section("location").
2. The tool sends a tiny message. The tool itself is a short Python function
marked with @function_tool. When the LLM calls it, it packs the section name into
a small JSON message — for example {"action": "show_section", "section": "hours"}
— and publishes it to the browser on a data channel labelled "ui".
3. The page listens and scrolls. In index.html, a few lines of
JavaScript listen for messages on the "ui" channel. When one arrives, the page
finds the section with that id, scrolls smoothly to it, and adds a gold outline so
the guest can see exactly which part Lin is talking about.
One detail matters: the three names hours, mains, and
location appear in both files — as a set called
SECTIONS in agent.py and as id attributes in
index.html. If you add a new section to the page, give the same name in both
places (and mention it in the prompt), or Lin will not be able to point at it.
Edit the knowledge without touching Python
Everything Lin knows lives in FAQs.txt, which sits next to agent.py:
Q: When is YouFantasy open?
A: YouFantasy is open every day from 9am to 10pm.
Q: What are the main courses?
A: Salmon, Beef, Chicken, Pork, Tofu.
Q: Where is YouFantasy?
A: YouFantasy is on Tat Chee Avenue, Kowloon Tong, Hong Kong.
Q: Any other question?
A: Sorry, I do not have answer for that question.
Every time a new call starts, agent.py reads this file fresh, replaces the
{faqs} placeholder in the prompt, and starts the session. So to teach Lin
something new, just add a Q&A pair, save the file, and open a new call. No Python edit,
no agent restart.
The files in the zip
pyproject.toml
[project]
name = "folder-ai-avatar-v3"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
"aiohttp>=3.14.4",
"livekit-agents[openai,silero]~=1.8",
"livekit-plugins-synthesia~=1.8",
"python-dotenv>=1.2.4",
]
[build-system]
requires = ["setuptools>=70"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
include = []
agent.py
import asyncio
import json
import os
from pathlib import Path
from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, function_tool, inference
from livekit.plugins import silero, synthesia
load_dotenv(dotenv_path=Path(__file__).parent / ".env")
AGENT_NAME = "avatar-demo"
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "8788bef1-8020-46e0-a8f4-510ea9989b25" # Jenny
VOICE_ID = "db6b0ed5-d5d3-463d-ae85-518a07d3c2b4" # Cartesia "Skylar"
FAQS_FILE = Path(__file__).parent / "FAQs.txt" # the avatar's knowledge
SECTIONS = {"hours", "mains", "location"} # must match the section ids in index.html
# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.
Answer only from these FAQs:
{faqs}
Those are the only things you know. If the guest asks anything the FAQs do not
cover, use the answer to "Any other question?" word for word.
The guest is looking at our web page. Every time you answer one of the FAQs,
also call show_section so the page scrolls to that part:
- opening hours: show_section("hours")
- main courses: show_section("mains")
- address: show_section("location")
Do not call show_section for any other question."""
class Host(Agent):
def __init__(self, room, instructions: str) -> None:
super().__init__(instructions=instructions)
self.room = room # needed to send messages to the browser
@function_tool
async def show_section(self, section: str) -> str:
"""Scroll the guest's web page to one section and highlight it.
Args:
section: One of: hours, mains, location.
"""
if section not in SECTIONS: # tell the model, so it retries with a real name
return f"No section {section!r}. Use one of: hours, mains, location."
message = json.dumps({"action": "show_section", "section": section})
await self.room.local_participant.publish_data(message, topic="ui") # index.html listens on "ui"
return "Done."
def prewarm(proc) -> None:
proc.userdata["vad"] = silero.VAD.load()
async def entrypoint(ctx: JobContext) -> None:
await ctx.connect()
# The brain: each line is one piece of the pipeline.
session = AgentSession(
vad=ctx.proc.userdata["vad"], # is someone talking?
stt=inference.STT(model="cartesia/ink-2"), # speech -> text
llm=inference.LLM(model="openai/gpt-4.1-mini"), # text -> reply (+ tool calls)
tts=inference.TTS(model="cartesia/sonic-3.6", voice=VOICE_ID), # reply -> speech
turn_handling={
"turn_detection": "stt", # have they finished?
"preemptive_generation": {"enabled": False}, # keep off when using tools
},
)
# The face. Start it BEFORE session.start().
avatar = synthesia.AvatarSession(
synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
join_timeout=60.0,
)
for attempt in range(3): # cold starts occasionally time out, so retry
try:
await avatar.start(session, room=ctx.room)
break
except synthesia.SynthesiaError as e:
if not e.retryable or attempt == 2:
raise
await asyncio.sleep(2)
# Read the FAQs fresh for each call, so edits to FAQs.txt show up on the next call.
faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
await session.start(agent=Host(ctx.room, INSTRUCTIONS.replace("{faqs}", faqs)), room=ctx.room)
session.generate_reply(
instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence."
)
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))
server.py
import os
import uuid
from datetime import timedelta
from pathlib import Path
from aiohttp import web
from dotenv import load_dotenv
from livekit import api
load_dotenv(dotenv_path=Path(__file__).parent / ".env")
AGENT_NAME = "avatar-demo"
async def token(request: web.Request) -> web.Response:
room = f"avatar-demo-{uuid.uuid4().hex[:8]}"
jwt = (
api.AccessToken()
.with_identity(f"user-{uuid.uuid4().hex[:8]}")
.with_ttl(timedelta(minutes=15))
.with_grants(api.VideoGrants(room_join=True, room=room))
.with_room_config(
api.RoomConfiguration(
sync_streams=True,
agents=[api.RoomAgentDispatch(agent_name=AGENT_NAME)],
)
)
.to_jwt()
)
return web.json_response({"url": os.environ["LIVEKIT_URL"], "token": jwt})
async def index(request: web.Request) -> web.FileResponse:
return web.FileResponse(
Path(__file__).parent / "index.html", headers={"Cache-Control": "no-store"}
)
app = web.Application()
app.router.add_get("/", index)
app.router.add_get("/token", token)
if __name__ == "__main__":
web.run_app(app, host="127.0.0.1", port=8080)
index.html
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>YouFantasy</title>
<style>
/* page styling omitted here for readability — see the zip for the full file */
html { scroll-behavior: smooth; }
section.highlight { box-shadow: 0 0 0 2px #c8a35a; } /* added by show_section */
#widget { position: fixed; right: 16px; bottom: 16px; width: 260px; }
</style>
</head>
<body>
<!-- The restaurant page. Each section id is a name show_section can use. -->
<nav>
<b>YouFantasy</b>
<a href="#hours">Hours</a>
<a href="#mains">Mains</a>
<a href="#location">Location</a>
</nav>
<section id="hours">
<h2>When we are open</h2>
<p>Every day, 9am to 10pm</p>
</section>
<section id="mains">
<h2>Main courses</h2>
<ul><li>Salmon</li><li>Beef</li><li>Chicken</li><li>Pork</li><li>Tofu</li></ul>
</section>
<section id="location">
<h2>Location</h2>
<p>Tat Chee Avenue, Kowloon Tong, Hong Kong</p>
</section>
<!-- The avatar -->
<div id="widget">
<video id="video" autoplay playsinline></video>
<span><span id="label">Talk to Lin</span><span id="mic"></span></span>
<button id="start">Start</button>
</div>
<script src="https://cdn.jsdelivr.net/npm/livekit-client@2/dist/livekit-client.umd.min.js"></script>
<script>
// What the page does when Lin calls show_section.
function showSection(section) {
const el = document.getElementById(section);
if (!el) return;
el.scrollIntoView({ behavior: 'smooth', block: 'start' });
document.querySelectorAll('.highlight').forEach((h) => h.classList.remove('highlight'));
el.classList.add('highlight');
}
const { Room, RoomEvent } = LivekitClient;
const video = document.getElementById('video');
const label = document.getElementById('label');
const btn = document.getElementById('start');
let room = null;
btn.onclick = async () => {
if (room) return leave();
btn.textContent = 'Stop'; label.textContent = 'Connecting…';
try {
const { url, token } = await (await fetch('/token')).json();
room = new Room();
room.on(RoomEvent.TrackSubscribed, (track) => track.attach(video));
room.on(RoomEvent.ParticipantAttributesChanged, (changed) => {
if (changed['lk.agent.state']) label.textContent = changed['lk.agent.state'];
});
// Tool calls arrive here as small JSON messages on the "ui" topic.
room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
if (topic !== 'ui') return;
const message = JSON.parse(new TextDecoder().decode(payload));
if (message.action === 'show_section') showSection(message.section);
});
await room.connect(url, token);
await room.localParticipant.setMicrophoneEnabled(true);
label.textContent = 'Waiting for Lin…';
} catch (err) { leave(); label.textContent = err.message; }
};
function leave() {
room?.disconnect(); room = null;
btn.textContent = 'Start'; label.textContent = 'Talk to Lin';
}
</script>
</body>
</html>
How it works, in plain words
Two programs, two terminals, one text file, one tool.
agent.py is the voice + avatar engine, and the only version-3 change inside it is
the show_section tool.
server.py is a tiny aiohttp web server that serves the page and mints a
short-lived LiveKit access token on each visit.
FAQs.txt is the knowledge base — the agent reads it fresh for every call
and injects its contents into the prompt as {faqs}.
The page is just a page. index.html is a normal restaurant website
with three sections marked by id attributes. The avatar widget floats in the corner,
on top of it. Nothing about the page is special — which is the point: you can drop the same
widget into any site you already have.
The LLM decides when to scroll. Nobody writes "if the question contains the word hours, scroll". The model reads the prompt's small rule book, understands the guest's question, and chooses to call the tool — the same way it chooses what words to say. If the question is not in the FAQs, it does not call the tool at all.
The message travels on a data channel. LiveKit rooms carry more than audio and
video; they also carry small text messages between participants. The agent publishes the
scroll instruction on a channel named "ui", and only the browser listens to it.
The whole trip — decision, message, scroll — finishes before Lin reaches the second
word of her answer.
Cold start retries. The Synthesia service occasionally takes a few seconds to spin up an avatar session, so the agent retries up to three times before giving up. That is why clicking Start can feel slow the first time — it is not broken, just warming up.
Version 2 vs. version 3 at a glance
| Version 2 | Version 3 | |
|---|---|---|
| Knowledge lives in | Text file (FAQs.txt) |
Text file (FAQs.txt) |
| Edit knowledge needs | Save file + new call | Save file + new call |
| The browser page | Bare video + Start button | A real web page with sections |
| Avatar can act on the page | No | Yes — scrolls and highlights (show_section tool) |
| Run commands | two terminals, same | two terminals, same |
Limitations (version 3)
- Lin still knows only what is in FAQs.txt. A plain text file is great for quick updates, but it cannot run calculations or look up real-time information. If you want weather, stock prices, or calendar booking, the next step is adding more tools or RAG.
- The scroll tool knows only three sections. To add a fourth one you must
edit three places in agreement: the prompt's rule book, the
SECTIONSset inagent.py, and a matchingidinindex.html. Miss one and the tool quietly does nothing (or the model retries with a valid name). - You need two accounts (LiveKit and Synthesia). LiveKit Cloud's Inference service covers STT, LLM, and TTS (including voices from Cartesia and Deepgram), so you do not need separate provider accounts. Synthesia offers a free Basic plan (at time of writing, Nov 2026) that includes Interactive Avatar with 1 concurrent session; paid plans raise that limit and add credits.
- Two-terminal workflow. You have to start
agent.pyandserver.pyseparately. That is fine for demos, but a production deployment would bundle them behind a single supervisor or run them on LiveKit Cloud Agents alone. - The web server has no authentication.
server.pygenerates a fresh room token for every visitor with no login. Fine for local testing; never deploy this as-is to the public internet.
Further reading
- livekit.io — the official LiveKit homepage with pricing and docs.
- livekit/agents — the LiveKit Agents Python repository with more examples including Synthesia.
- LiveKit Synthesia Avatar docs — the official guide for wiring a Synthesia avatar into LiveKit Agents.
- synthesia.io — create and manage your avatars.
- How to use LiveKit and Synthesia Avatar (version 2)?
— the previous version, where the knowledge moved to
FAQs.txtbut the page could not react yet. - How to use LiveKit and Synthesia Avatar (version 1)? — the simpler first version of this same avatar, with knowledge hardcoded in Python.
- How to use LiveKit Voice Agent with Weather Tool? — another example of giving an agent a tool, this time to call an external API.