English 繁體中文 简体中文

How to use LiveKit Voice Agent with Weather Tool?

Howard Pong Yuen Lam · AI Notes · November 4, 2026 · 6 min read

A voice agent is an AI program that you can talk to in real time: you speak, it recognises your words (STT, speech-to-text), sends the text to a large language model (LLM), then turns the answer back into speech (TTS, text-to-speech). LiveKit is an open-source framework that ties these three pieces together over the internet, so your agent can run on a server and answer calls from a browser or a phone.

This note walks you through a minimal but complete voice agent in Python. The program greets you, tells you the weather (a demo tool).

Watch the walkthrough

Open in Google Drive (no account needed).

What you need

Set up LiveKit Cloud

Sign up at cloud.livekit.io. In the dashboard, click Projects → Create Project, then open the project and go to Keys. Copy the three values:

In your project folder, create a file named .env with these three lines:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=APIxxxxxxxxxxxxx
LIVEKIT_API_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Important: never put .env on GitHub or any public site — it contains your live credentials.

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the program and a pyproject.toml project file. Add your own .env file with the keys, as shown above. Then open the folder in Visual Studio Code and type these two lines in the terminal, one at a time:

uv sync
uv run main.py console

uv sync reads pyproject.toml, downloads Python 3.13 if you do not already have it, and installs every package the agent needs. uv run main.py console then starts the agent in your Terminal so you can speak to it directly. You only need to run uv sync once — only re-run it if you later change main.py or the dependencies.

The Python files

The zip contains two files: pyproject.toml and main.py. Here they are for reference.

pyproject.toml

[project]
name = "folder-ai-livekit-agent"
version = "0.1.0"
requires-python = ">=3.13"
dependencies = [
    "livekit-agents>=1.8.4,<2",
    "python-dotenv",
    "requests",
]

[build-system]
requires = ["uv_build>=0.13.6,<0.14.0"]
build-backend = "uv_build"

main.py

import requests
from dotenv import load_dotenv
from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    RunContext,
    cli,
    function_tool,
    inference,
)

load_dotenv()


def _geocode(city: str) -> tuple[float, float, str] | None:
    """Resolve a city name to (lat, lon, resolved_label) using Open-Meteo geocoding."""
    r = requests.get(
        "https://geocoding-api.open-meteo.com/v1/search",
        params={"name": city, "count": 1, "language": "en", "format": "json"},
        timeout=10,
    )
    results = r.json().get("results")
    if not results:
        return None
    best = results[0]
    name = best.get("name", city)
    country = best.get("country")
    label = f"{name}, {country}" if country else name
    return best["latitude"], best["longitude"], label


def _ip_location() -> tuple[float, float, str] | None:
    """Resolve lat/lon AND a human-readable city/region label from the caller's IP."""
    try:
        info = requests.get("https://ipinfo.io/json", timeout=10).json()
        lat_str, lon_str = info["loc"].split(",")
        parts = [p for p in (info.get("city"), info.get("region"), info.get("country")) if p]
        label = ", ".join(parts) if parts else "your current location"
        return float(lat_str), float(lon_str), label
    except Exception:
        return None


def _fetch_weather(latitude: float, longitude: float) -> dict:
    params = {
        "latitude": latitude,
        "longitude": longitude,
        "current": "temperature_2m,weather_code,relative_humidity_2m,apparent_temperature",
        "daily": "temperature_2m_max,temperature_2m_min,weather_code",
        "temperature_unit": "celsius",
        "timezone": "auto",
        "forecast_days": 1,
    }
    data = requests.get(
        "https://api.open-meteo.com/v1/forecast",
        params=params,
        timeout=10,
    ).json()
    return data


# WMO weather code -> short English description
_WMO_CODES = {
    0: "clear sky",
    1: "mainly clear", 2: "partly cloudy", 3: "overcast",
    45: "fog", 48: "freezing fog",
    51: "light drizzle", 53: "moderate drizzle", 55: "dense drizzle",
    56: "light freezing drizzle", 57: "dense freezing drizzle",
    61: "slight rain", 63: "moderate rain", 65: "heavy rain",
    66: "light freezing rain", 67: "heavy freezing rain",
    71: "slight snow fall", 73: "moderate snow fall", 75: "heavy snow fall",
    77: "snow grains",
    80: "slight rain showers", 81: "moderate rain showers", 82: "violent rain showers",
    85: "slight snow showers", 86: "heavy snow showers",
    95: "thunderstorm", 96: "thunderstorm with slight hail", 99: "thunderstorm with heavy hail",
}


@function_tool
async def lookup_weather(
    context: RunContext,
    location: str = "",
):
    """
    Look up weather information. If the user mentions a specific city or
    place (e.g. "Tokyo", "Mountain View"), pass it in. Otherwise - when the
    user says "here", "my city", "where I am", or does not name any place -
    call this tool with no location argument. It will automatically detect
    the user's approximate city from their IP address.
    """

    lat = lon = None
    resolved = ""

    if location and location.strip():
        geo = _geocode(location.strip())
        if geo:
            lat, lon, resolved = geo

    if lat is None:
        ip = _ip_location()
        if ip is None:
            return {"error": "Could not detect your location automatically."}
        lat, lon, resolved = ip

    try:
        data = _fetch_weather(lat, lon)
    except Exception as e:
        return {"error": f"Weather service unavailable: {e}"}

    cur = data.get("current", {}) or {}
    daily = data.get("daily", {}) or {}
    code = cur.get("weather_code")
    condition = _WMO_CODES.get(code, f"code {code}") if code is not None else "unknown"

    return {
        "location": resolved,
        "temperature_c": cur.get("temperature_2m"),
        "feels_like_c": cur.get("apparent_temperature"),
        "condition": condition,
        "humidity_percent": cur.get("relative_humidity_2m"),
        "today_high_c": (daily.get("temperature_2m_max") or [None])[0],
        "today_low_c": (daily.get("temperature_2m_min") or [None])[0],
    }


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )

    agent = Agent(
        instructions=(
            "You are a friendly voice assistant built by LiveKit. "
            "You can answer weather questions. You do NOT need to ask the "
            "user which city they are in - if they do not name a specific "
            "city, just call lookup_weather() with no arguments and it will "
            "auto-detect their location. Only pass a location when the user "
            "explicitly names one (e.g. 'what's the weather in Tokyo?'). "
            "Never ask 'which city?' - always try to answer directly first."
        ),
        tools=[lookup_weather],
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")


if __name__ == "__main__":
    cli.run_app(server)

How main.py works, in plain words

Voice Activity Detection, STT, LLM, TTS. The AgentSession line wires four pieces together. vad=inference.VAD() is Voice Activity Detection — it listens to the microphone stream and tells the agent when you start and stop talking, so the LLM only processes real speech instead of background noise. Speech is turned to text by Deepgram (deepgram/nova-3), text is answered by Google Gemma 4 (google/gemma-4-31b-it), and the answer is spoken by Cartesia (cartesia/sonic-3). All four run on LiveKit Cloud so you do not need separate API keys for each provider — a single LiveKit account covers them.

A real weather tool. The lookup_weather function is wrapped with @function_tool, which tells the LLM “you can call this function when the user asks about the weather.” The function is fully working: it calls two free Open-Meteo endpoints (one for city name → lat/lon geocoding, one for the actual forecast) and falls back to ipinfo.io to guess the caller's approximate city from their IP when no place is named. It returns temperature (current + today's high/low in Celsius), feels-like temperature, humidity, a condition description (mapped from WMO weather codes like 65 → "heavy rain"), and the resolved city name. If any API fails, it returns an "error" key so the LLM can apologise instead of fabricating numbers.

Prompt engineering saves a tool call. The agent instructions are deliberately long: “You do NOT need to ask the user which city they are in — if they do not name a specific city, just call lookup_weather() with no arguments and it will auto-detect their location. Never ask ‘which city?’ — always try to answer directly first.” Without this paragraph, the LLM would often waste a turn asking for clarification, which sounds robotic and slows the conversation.

A friendly greeting. After the session starts, generate_reply(instructions="greet the user and ask about their day") makes the agent speak the first thing, so callers hear a hello instead of silence.

Talk to your agent

With uv run main.py console running in your Terminal, the program will ask your Mac for microphone permission — click Allow. Then speak into your microphone: you should hear the agent greet you back, and you can continue the conversation freely. Press Ctrl+C when you want to stop.

If the Terminal does not print anything after you press play, or it shows an error about missing credentials, stop the server (Ctrl+C), check that .env is in the project folder, and run uv run main.py console again.

Limitations

Further reading