English 繁體中文 简体中文

How to use LiveKit Voice Agent to Chat?

Howard Pong Yuen Lam · AI Notes · November 4, 2026 · 5 min read

A voice agent is an AI program that you can talk to in real time: you speak, it recognises your words (STT, speech-to-text), sends the text to a large language model (LLM), then turns the answer back into speech (TTS, text-to-speech). LiveKit is an open-source framework that ties these three pieces together over the internet, so your agent can run on a server and answer calls from a browser or a phone.

This note walks you through a minimal but complete voice agent in Python. The program greets you and answers any question using the LLM — no custom tools required.

Watch the walkthrough

Open in Google Drive (no account needed).

What you need

Set up LiveKit Cloud

Sign up at cloud.livekit.io. In the dashboard, click Projects → Create Project, then open the project and go to Keys. Copy the three values:

In your project folder, create a file named .env with these three lines:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=APIxxxxxxxxxxxxx
LIVEKIT_API_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Important: never put .env on GitHub or any public site — it contains your live credentials.

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the program and a pyproject.toml project file. Add your own .env file with the keys, as shown above. Then open the folder in Visual Studio Code and type these two lines in the terminal, one at a time:

uv sync
uv run main.py console

uv sync reads pyproject.toml, downloads Python 3.13 if you do not already have it, and installs every package the agent needs. uv run main.py console then starts the agent in your Terminal so you can speak to it directly. You only need to run uv sync once — only re-run it if you later change main.py or the dependencies.

The Python files

The zip contains two files: pyproject.toml and main.py. Here they are for reference.

pyproject.toml

[project]
name = "folder-ai-livekit-agent"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "livekit-agents>=1.8.4,<2",
    "python-dotenv",
]

[build-system]
requires = ["uv_build>=0.13.6,<0.14.0"]
build-backend = "uv_build"

main.py

from dotenv import load_dotenv
from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    cli,
    inference,
)

load_dotenv()


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )

    agent = Agent(
        instructions=(
            "You are a helpful and friendly voice assistant built by LiveKit. "
            "Use your knowledge to answer any question the user asks, clearly "
            "and concisely. Engage naturally in conversation."
        ),
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask what they'd like to talk about")


if __name__ == "__main__":
    cli.run_app(server)

How main.py works, in plain words

Voice Activity Detection, STT, LLM, TTS. The AgentSession line wires four pieces together. vad=inference.VAD() is Voice Activity Detection — it listens to the microphone stream and tells the agent when you start and stop talking, so the LLM only processes real speech instead of background noise. Speech is turned to text by Deepgram (deepgram/nova-3), text is answered by Google Gemma 4 (google/gemma-4-31b-it), and the answer is spoken by Cartesia (cartesia/sonic-3). All four run on LiveKit Cloud so you do not need separate API keys for each provider — a single LiveKit account covers them.

No tools needed. Unlike agents that call external APIs (weather, calendar, databases), this agent answers directly from the LLM's built-in knowledge. That means zero setup beyond LiveKit Cloud credentials and zero external dependencies — but also no access to real-time or private data. Tools are optional; you add them only when you need them.

A friendly greeting. After the session starts, generate_reply(instructions="greet the user and ask what they'd like to talk about") makes the agent speak the first thing, so callers hear a hello instead of silence.

Talk to your agent

With uv run main.py console running in your Terminal, the program will ask your Mac for microphone permission — click Allow. Then speak into your microphone: you should hear the agent greet you back, and you can continue the conversation freely. Press Ctrl+C when you want to stop.

If the Terminal does not print anything after you press play, or it shows an error about missing credentials, stop the server (Ctrl+C), check that .env is in the project folder, and run uv run main.py console again.

Limitations

Further reading