English 繁體中文 简体中文

How to use LiveKit and Synthesia Avatar (version 1)?

Howard Pong Yuen Lam · AI Notes · November 4, 2026 · 6 min read

A voice agent that answers only with text or a disembodied voice is useful — but an agent with a face feels much more like talking to a real person. Synthesia makes photorealistic AI avatars that speak and lip-sync in real time. LiveKit ties the avatar to the voice agent pipeline: microphone → speech recognition → LLM → text-to-speech → avatar.

This note shows you the smallest complete demo: a browser page with a video window, a single button, and an avatar called Lin who plays the host at a fictional restaurant and answers when it opens. It is version 1 — deliberately minimal so every line is easy to follow.

Watch the walkthrough

Open in Google Drive (no account needed).

The pipeline

browser mic → VAD → STT → LLM → TTS → Synthesia avatar → browser video

Five pieces wired together over LiveKit's real-time transport. VAD (Voice Activity Detection) listens for when you start and stop talking, STT turns speech to text, LLM answers, TTS turns the answer in text back to speech, and Synthesia renders the face with lips that match the audio. All five run on LiveKit Cloud — no separate servers needed.

What you need

Set up LiveKit Cloud

Sign up at cloud.livekit.io. In the dashboard, click Projects → Create Project, then open the project and go to Keys. Copy the three values:

Set up Synthesia

Sign in at app.synthesia.io. Go to Avatars and pick or create an avatar — each one has a UUID, shown in the URL when you open it (it looks like 3190429b-1eda-4151-bf24-96162e405840). Copy that as SYNTHESIA_AVATAR_ID.

Then go to API Keys and create a key — that is SYNTHESIA_API_KEY.

Create your .env

In your project folder, create a file named .env with these five lines:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=APIxxxxxxxxxxxxx
LIVEKIT_API_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SYNTHESIA_API_KEY=syn_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SYNTHESIA_AVATAR_ID=3190429b-1eda-4151-bf24-96162e405840

Important: never put .env on GitHub or any public site — it contains your live credentials.

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds agent.py, server.py, index.html, and pyproject.toml. Add your own .env file with the keys, as shown above. Then open the folder in Visual Studio Code and run these two commands in two separate Terminal windows:

Terminal 1 — the voice agent + avatar engine:

uv sync
uv run agent.py dev

Terminal 2 — the web server that serves the browser page:

uv run server.py

uv sync only needs to run the first time — it installs Python 3.13 and every package. After that you just start the agent in one terminal and the server in the other.

Open the page in your browser

With both terminals running, open http://127.0.0.1:8080 in Chrome, Safari, or Edge. Click Start, allow microphone permission, and wait a few seconds while the agent and avatar join the room. You should see a video of Lin greeting you — click the microphone icon and ask a question, for example "When are you open?"

The files in the zip

pyproject.toml

[project]
name = "folder-ai-avatar"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "aiohttp>=3.14.4",
    "livekit-agents[openai,silero]~=1.8",
    "livekit-plugins-synthesia~=1.8",
    "python-dotenv>=1.2.4",
]

[build-system]
requires = ["setuptools>=70"]
build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
include = []

agent.py

import asyncio
import os
from pathlib import Path

from dotenv import load_dotenv
from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli, inference
from livekit.plugins import silero, synthesia

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"
AVATAR_ID = os.getenv("SYNTHESIA_AVATAR_ID") or "3190429b-1eda-4151-bf24-96162e405840"
VOICE_ID = "db6b0ed5-d5d3-463d-ae85-518a07d3c2b4"

INSTRUCTIONS = """\
You are Lin, the host at YouFantasy, a made-up restaurant. You are a real-time
avatar in a spoken conversation, so answer in one short sentence and never use
markdown, lists, or URLs.

YouFantasy is open every day from 9am to 10pm.

That is the only thing you know. If the guest asks about anything else, say
kindly that for now you can only help with our opening hours."""


def prewarm(proc) -> None:
    proc.userdata["vad"] = silero.VAD.load()


async def entrypoint(ctx: JobContext) -> None:
    await ctx.connect()

    session = AgentSession(
        vad=ctx.proc.userdata["vad"],
        stt=inference.STT(model="cartesia/ink-2"),
        llm=inference.LLM(model="openai/gpt-4.1-mini"),
        tts=inference.TTS(model="cartesia/sonic-3.6", voice=VOICE_ID),
        turn_handling={"turn_detection": "stt"},
    )

    avatar = synthesia.AvatarSession(
        synthesia.AvatarConfig(avatar_ids=[AVATAR_ID]),
        join_timeout=60.0,
    )
    for attempt in range(3):
        try:
            await avatar.start(session, room=ctx.room)
            break
        except synthesia.SynthesiaError as e:
            if not e.retryable or attempt == 2:
                raise
            await asyncio.sleep(2)

    await session.start(agent=Agent(instructions=INSTRUCTIONS), room=ctx.room)
    session.generate_reply(
        instructions="Welcome the guest to YouFantasy and invite them to ask about our opening hours, in one short sentence."
    )


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name=AGENT_NAME, port=8081))

server.py

import os
import uuid
from datetime import timedelta
from pathlib import Path

from aiohttp import web
from dotenv import load_dotenv
from livekit import api

load_dotenv(dotenv_path=Path(__file__).parent / ".env")

AGENT_NAME = "avatar-demo"


async def token(request: web.Request) -> web.Response:
    room = f"avatar-demo-{uuid.uuid4().hex[:8]}"
    jwt = (
        api.AccessToken()
        .with_identity(f"user-{uuid.uuid4().hex[:8]}")
        .with_ttl(timedelta(minutes=15))
        .with_grants(api.VideoGrants(room_join=True, room=room))
        .with_room_config(
            api.RoomConfiguration(
                sync_streams=True,
                agents=[api.RoomAgentDispatch(agent_name=AGENT_NAME)],
            )
        )
        .to_jwt()
    )
    return web.json_response({"url": os.environ["LIVEKIT_URL"], "token": jwt})


async def index(request: web.Request) -> web.FileResponse:
    return web.FileResponse(
        Path(__file__).parent / "index.html", headers={"Cache-Control": "no-store"}
    )


app = web.Application()
app.router.add_get("/", index)
app.router.add_get("/token", token)

if __name__ == "__main__":
    web.run_app(app, host="127.0.0.1", port=8080)

index.html

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>YouFantasy · Step 01</title>
  <style>
    body { margin: 0; min-height: 100vh; display: grid; place-items: center;
           background: #0b0a09; color: #f4efe6; font-family: ui-sans-serif, system-ui, sans-serif; }
    main { width: min(420px, 100% - 32px); text-align: center; }
    h1 { font-family: Georgia, serif; font-weight: 400; margin: 0 0 4px; }
    video { width: 100%; aspect-ratio: 1; object-fit: cover; background: #141210;
            border: 1px solid #2a2521; border-radius: 12px; }
    #mic { display: inline-block; width: 8px; height: 8px; border-radius: 50%;
           background: #2a2521; margin-left: 8px; }
    #mic.on { background: #46d17f; }
    button { background: #c8a35a; color: #0b0a09; border: 0; border-radius: 6px;
             padding: 8px 16px; font-weight: 600; cursor: pointer; }
  </style>
</head>
<body>
  <main>
    <h1>YouFantasy</h1>
    <p>Ask Lin when we are open.</p>
    <video id="video" autoplay playsinline></video>
    <div>
      <span><span id="label">Talk to Lin</span><span id="mic"></span></span>
      <button id="start">Start</button>
    </div>
  </main>
  <script src="https://cdn.jsdelivr.net/npm/livekit-client@2/dist/livekit-client.umd.min.js"></script>
  <script>
    const { Room, RoomEvent } = LivekitClient;
    const video = document.getElementById('video');
    const label = document.getElementById('label');
    const mic = document.getElementById('mic');
    const btn = document.getElementById('start');
    let room = null;

    btn.onclick = async () => {
      if (room) return leave();
      btn.textContent = 'Stop'; label.textContent = 'Connecting…';
      try {
        const { url, token } = await (await fetch('/token')).json();
        room = new Room();
        room.on(RoomEvent.TrackSubscribed, (track) => track.attach(video));
        room.on(RoomEvent.ParticipantAttributesChanged, (changed) => {
          if (changed['lk.agent.state']) label.textContent = changed['lk.agent.state'];
        });
        room.on(RoomEvent.ActiveSpeakersChanged, (speakers) =>
          mic.classList.toggle('on', speakers.some((s) => s.isLocal)));
        room.on(RoomEvent.Disconnected, leave);
        await room.connect(url, token);
        await room.localParticipant.setMicrophoneEnabled(true);
        label.textContent = 'Waiting for Lin…';
      } catch (err) { leave(); label.textContent = err.message; }
    };

    function leave() {
      room?.disconnect(); room = null; mic.classList.remove('on');
      btn.textContent = 'Start'; label.textContent = 'Talk to Lin';
    }
  </script>
</body>
</html>

How it works, in plain words

Two programs, two terminals. agent.py is the voice + avatar engine — it joins a LiveKit room, listens to the microphone, answers, and drives the avatar. server.py is a tiny aiohttp web server that serves the page and, on each visit, mints a short-lived LiveKit access token (so you do not embed credentials in HTML). You start both separately because they each own their own port.

The browser knows nothing about avatars. index.html just joins a LiveKit room and plays every video track that arrives — it does not care whether the track comes from a real webcam or a Synthesia avatar. The avatar streams itself from inside the agent, and sync_streams=True in the room config keeps its lips aligned with the audio track.

Cold start retries. The Synthesia service occasionally takes a few seconds to spin up an avatar session, so the agent retries up to three times before giving up. That is why clicking Start can feel slow the first time — it is not broken, just warming up.

Limitations (version 1)

Further reading