How to use LiveKit and Synthesia Avatar (version 2)?
Synthesia makes photorealistic AI avatars that speak and lip-sync in real time. LiveKit ties the avatar to a voice agent: microphone → speech recognition → LLM → text-to-speech → avatar.
In version 1, everything the avatar knew was hardcoded in the Python prompt. That makes the code easy to follow, but every time you want Lin to know something new you have to edit the Python file and restart it.
Version 2 moves the knowledge to a plain text file called FAQs.txt for frequently asked questions. Edit the
file, open a new call, and the avatar picks up the change — no code edit. Lin now answers four questions (opening hours, main courses, location, and a
fallback) instead of one.
The pipeline
browser mic → VAD → STT → turn detection → LLM → TTS → Synthesia avatar → browser video
Five pieces wired together over LiveKit's real-time transport. VAD (Voice Activity Detection) listens for when you start and stop talking, STT turns speech to text, LLM answers, TTS turns the answer in text back to speech, and Synthesia renders the face with lips that match the audio. All five run on LiveKit Cloud.
What you need
- A Mac (the same steps work on Windows and Linux too).
- Homebrew installed and the
uvpackage manager — if you do not haveuv, runbrew install uv. - A LiveKit Cloud account (the free Build plan is enough).
- A Synthesia account. You need at least one avatar created there (the free plan lets you try).
- Your LiveKit and Synthesia credentials, which you copy into a file called
.env— see below.
Set up LiveKit Cloud
Sign up at cloud.livekit.io. In the dashboard, click Projects → Create Project, then open the project and go to Keys. Copy the three values:
LIVEKIT_URL— looks likewss://your-project.livekit.cloud.LIVEKIT_API_KEY— a short string likeAPIxxxxxxxxxxxxx.LIVEKIT_API_SECRET— a longer string, shown only once.
Set up Synthesia
Sign in at app.synthesia.io.
Go to Avatars and pick or create an avatar — each one has a UUID, shown in the
URL when you open it (it looks like 3190429b-1eda-4151-bf24-96162e405840).
Copy that as SYNTHESIA_AVATAR_ID.
Then go to API Keys
and create a key — that is SYNTHESIA_API_KEY.
Create your .env
In your project folder, create a file named .env with these five lines:
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=APIxxxxxxxxxxxxx
LIVEKIT_API_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SYNTHESIA_API_KEY=syn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SYNTHESIA_AVATAR_ID=3190429b-1eda-4151-bf24-96162e405840
Important: never put .env on GitHub or any public site — it contains your live credentials.
Running it on your own computer
Download the zip file and
unzip it to a folder. The zip holds agent.py, server.py,
index.html, FAQs.txt, pyproject.toml, and README.md.
Add your own .env file with the keys, as shown above. Then open the folder in
Visual Studio Code and run these two commands in two separate Terminal windows:
Terminal 1 — the voice agent + avatar engine:
uv sync
uv run agent.py dev
Terminal 2 — the web server that serves the browser page:
uv run server.py
uv sync only needs to run the first time — it installs Python 3.13 and every
package. After that you just start the agent in one terminal and the server in the other.
Open the page in your browser
With both terminals running, open http://127.0.0.1:8080 in Chrome, Safari, or Edge. Click Start, allow microphone permission, and wait a few seconds while the agent and avatar join the room. You should see a video of Lin greeting you — click the microphone icon and ask a question, for example "Where are you located?" or "What main courses do you serve?"
Edit the knowledge without touching Python
Everything Lin knows lives in FAQs.txt, which sits next to agent.py:
Q: When is YouFantasy open?
A: YouFantasy is open every day from 9am to 10pm.
Q: What are the main courses?
A: Salmon, Beef, Chicken, Pork, Tofu.
Q: Where is YouFantasy?
A: YouFantasy is on Tat Chee Avenue, Kowloon Tong, Hong Kong.
Q: Any other question?
A: Sorry, I do not have answer for that question.
Every time a new call starts, agent.py reads this file fresh, replaces the
{faqs} placeholder in the prompt, and starts the session. So to teach Lin
something new, just add a Q&A pair, save the file, and open a new call. No Python edit,
no agent restart.
The files in the zip
pyproject.toml
[project]
name = "folder-ai-avatar-v2"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
"aiohttp>=3.14.4",
"livekit-agents[openai,silero]~=1.8",
"livekit-plugins-synthesia~=1.8",
"python-dotenv>=1.2.4",
]
[build-system]
requires = ["setuptools>=70"]
build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
include = []
agent.py
import asyncio
import os
from pathlib import Path
from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, inference
from livekit.plugins import silero, synthesia
load_dotenv(dotenv_path=Path(__file__).parent / ".env")
AGENT_NAME = "avatar-demo"
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "3190429b-1eda-4151-bf24-96162e405840"
VOICE_ID = "db6b0ed5-d5d3-463d-ae85-518a07d3c2b4"
FAQS_FILE = Path(__file__).parent / "FAQs.txt" # the avatar's knowledge
# {faqs} is replaced with the contents of FAQs.txt at the start of every call.
INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one to three short sentences and never use
markdown, lists, or URLs.
Answer only from these FAQs:
{faqs}
Those are the only things you know. If the guest asks anything the FAQs do not
cover, use the answer to "Any other question?" word for word."""
def prewarm(proc) -> None:
proc.userdata["vad"] = silero.VAD.load()
async def entrypoint(ctx: JobContext) -> None:
await ctx.connect()
session = AgentSession(
vad=ctx.proc.userdata["vad"],
stt=inference.STT(model="cartesia/ink-2"),
llm=inference.LLM(model="openai/gpt-4.1-mini"),
tts=inference.TTS(model="cartesia/sonic-3.6", voice=VOICE_ID),
turn_handling={"turn_detection": "stt"},
)
avatar = synthesia.AvatarSession(
synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
join_timeout=60.0,
)
for attempt in range(3):
try:
await avatar.start(session, room=ctx.room)
break
except synthesia.SynthesiaError as e:
if not e.retryable or attempt == 2:
raise
await asyncio.sleep(2)
# Read FAQs fresh each call so edits to FAQs.txt show up on the next call.
faqs = FAQS_FILE.read_text(encoding="utf-8").strip()
await session.start(
agent=Agent(instructions=INSTRUCTIONS.replace("{faqs}", faqs)),
room=ctx.room,
)
session.generate_reply(
instructions="Welcome the guest to YouFantasy and invite them to ask a question, in one short sentence."
)
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))
server.py
import os
import uuid
from datetime import timedelta
from pathlib import Path
from aiohttp import web
from dotenv import load_dotenv
from livekit import api
load_dotenv(dotenv_path=Path(__file__).parent / ".env")
AGENT_NAME = "avatar-demo"
async def token(request: web.Request) -> web.Response:
room = f"avatar-demo-{uuid.uuid4().hex[:8]}"
jwt = (
api.AccessToken()
.with_identity(f"user-{uuid.uuid4().hex[:8]}")
.with_ttl(timedelta(minutes=15))
.with_grants(api.VideoGrants(room_join=True, room=room))
.with_room_config(
api.RoomConfiguration(
sync_streams=True,
agents=[api.RoomAgentDispatch(agent_name=AGENT_NAME)],
)
)
.to_jwt()
)
return web.json_response({"url": os.environ["LIVEKIT_URL"], "token": jwt})
async def index(request: web.Request) -> web.FileResponse:
return web.FileResponse(
Path(__file__).parent / "index.html", headers={"Cache-Control": "no-store"}
)
app = web.Application()
app.router.add_get("/", index)
app.router.add_get("/token", token)
if __name__ == "__main__":
web.run_app(app, host="127.0.0.1", port=8080)
index.html
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>YouFantasy</title>
<style>
body { margin: 0; min-height: 100vh; display: grid; place-items: center;
background: #0b0a09; color: #f4efe6; font-family: ui-sans-serif, system-ui, sans-serif; }
main { width: min(420px, 100% - 32px); text-align: center; }
h1 { font-family: Georgia, serif; font-weight: 400; margin: 0 0 4px; }
video { width: 100%; aspect-ratio: 1; object-fit: cover; background: #141210;
border: 1px solid #2a2521; border-radius: 12px; }
#mic { display: inline-block; width: 8px; height: 8px; border-radius: 50%;
background: #2a2521; margin-left: 8px; }
#mic.on { background: #46d17f; }
button { background: #c8a35a; color: #0b0a09; border: 0; border-radius: 6px;
padding: 8px 16px; font-weight: 600; cursor: pointer; }
</style>
</head>
<body>
<main>
<h1>YouFantasy</h1>
<p>Ask Lin a question.</p>
<video id="video" autoplay playsinline></video>
<div>
<span><span id="label">Talk to Lin</span><span id="mic"></span></span>
<button id="start">Start</button>
</div>
</main>
<script src="https://cdn.jsdelivr.net/npm/livekit-client@2/dist/livekit-client.umd.min.js"></script>
<script>
const { Room, RoomEvent } = LivekitClient;
const video = document.getElementById('video');
const label = document.getElementById('label');
const mic = document.getElementById('mic');
const btn = document.getElementById('start');
let room = null;
btn.onclick = async () => {
if (room) return leave();
btn.textContent = 'Stop'; label.textContent = 'Connecting…';
try {
const { url, token } = await (await fetch('/token')).json();
room = new Room();
room.on(RoomEvent.TrackSubscribed, (track) => track.attach(video));
room.on(RoomEvent.ParticipantAttributesChanged, (changed) => {
if (changed['lk.agent.state']) label.textContent = changed['lk.agent.state'];
});
room.on(RoomEvent.ActiveSpeakersChanged, (speakers) =>
mic.classList.toggle('on', speakers.some((s) => s.isLocal)));
room.on(RoomEvent.Disconnected, leave);
await room.connect(url, token);
await room.localParticipant.setMicrophoneEnabled(true);
label.textContent = 'Waiting for Lin…';
} catch (err) { leave(); label.textContent = err.message; }
};
function leave() {
room?.disconnect(); room = null; mic.classList.remove('on');
btn.textContent = 'Start'; label.textContent = 'Talk to Lin';
}
</script>
</body>
</html>
How it works, in plain words
Two programs, two terminals, one text file.
agent.py is the voice + avatar engine.
server.py is a tiny aiohttp web server that serves the page and mints a
short-lived LiveKit access token on each visit.
FAQs.txt is the knowledge base — the agent reads it fresh for every call
and injects its contents into the prompt as {faqs}.
Why a text file? A non-programmer can maintain it. A restaurant manager, a
student writing a project, a teacher building a demo — none of them need to open Python
to add a fact. The only rule is the Q/A format: each pair starts with Q: and
A: on separate lines, separated from the next pair by a blank line.
The browser knows nothing about avatars. index.html just joins
a LiveKit room and plays every video track that arrives — it does not care whether the track
comes from a real webcam or a Synthesia avatar. The avatar streams itself from inside the
agent, and sync_streams=True in the room config keeps its lips aligned with the
audio track.
Cold start retries. The Synthesia service occasionally takes a few seconds to spin up an avatar session, so the agent retries up to three times before giving up. That is why clicking Start can feel slow the first time — it is not broken, just warming up.
Version 1 vs. version 2 at a glance
| Version 1 | Version 2 | |
|---|---|---|
| Knowledge lives in | Python prompt (INSTRUCTIONS) |
Text file (FAQs.txt) |
| Edit knowledge needs | Python edit + restart | Save file + new call |
| Question count | 1 (opening hours) | 4 (hours, courses, location, fallback) |
| Run commands | two terminals, same | two terminals, same |
Limitations (version 2)
- Lin still knows only what is in FAQs.txt. A plain text file is great for quick updates, but it cannot run calculations or look up real-time information. If you want weather, stock prices, or calendar booking, the next step is adding tools or RAG.
- You need two accounts (LiveKit and Synthesia). LiveKit Cloud's Inference service covers STT, LLM, and TTS (including voices from Cartesia and Deepgram), so you do not need separate provider accounts. Synthesia offers a free Basic plan (at time of writing, Nov 2026) that includes Interactive Avatar with 1 concurrent session; paid plans raise that limit and add credits.
- Two-terminal workflow. You have to start
agent.pyandserver.pyseparately. That is fine for demos, but a production deployment would bundle them behind a single supervisor or run them on LiveKit Cloud Agents alone. - The web server has no authentication.
server.pygenerates a fresh room token for every visitor with no login. Fine for local testing; never deploy this as-is to the public internet.
Further reading
- livekit.io — the official LiveKit homepage with pricing and docs.
- livekit/agents — the LiveKit Agents Python repository with more examples including Synthesia.
- LiveKit Synthesia Avatar docs — the official guide for wiring a Synthesia avatar into LiveKit Agents.
- synthesia.io — create and manage your avatars.
- How to use LiveKit and Synthesia Avatar (version 1)? — the simpler first version of this same avatar, with knowledge hardcoded in Python.
- How to use LiveKit Voice Agent to Chat? — the chat-only version of this agent, without the avatar face.
- How to use LiveKit Voice Agent with Weather Tool? — a voice agent that calls external APIs as tools.