How to create a web scraping tool for an AI agent?

Howard Pong Yuen Lam · AI Notes · November 15, 2026 · 4 min read

English 繁體中文 简体中文

An AI agent can plan, choose a function and use the answer. It cannot open a webpage by itself. If you ask about a page it has never read, it either says it does not know or invents something that sounds right.

A tool is how you close that gap: an ordinary Python function the agent is allowed to call. This note builds the scraping tool from the agent toolkit. It downloads a page, keeps the readable text, saves that text to a file, and returns the name of the file. The whole tool is 31 lines.

Step 1: The function and its description

def scrape_webpage(url: str, file_path: str) -> str:
    """
    Fetch a webpage and save its visible text content into a .txt file.

    Args:
        url (str): The webpage URL to scrape.
        file_path (str): Destination path for the .txt file.

    Returns:
        str: The file path that was written.
    """

The first line names the function, scrape_webpage, and lists what it needs: a webpage address in url and a destination in file_path. The -> str at the end says the function gives back text, in this case the name of the file it wrote.

The text between the triple quotation marks is a docstring. It is not a comment. When this function is offered to an agent, the docstring becomes the description the model reads before deciding whether to call it. It says what the tool does, what the two inputs are, and what comes back, which is exactly what a model needs in order to use the tool correctly.

Step 2: Downloading the page

response = requests.get(url, timeout=10)

The requests package downloads pages from the internet. requests.get asks the website for the page at url, and response holds the answer. timeout=10 gives up after ten seconds instead of waiting forever. If the address is wrong or the site refuses, requests stops the program with an error rather than returning an empty page.

Step 3: Keeping only the readable text

soup = BeautifulSoup(response.text, "html.parser")
text = soup.get_text(separator="\n", strip=True)

What came back is HTML, the code that describes a page. BeautifulSoup, from the bs4 package, reads that code into a structure the program can inspect; the result is named soup. "html.parser" is the parser built into Python, so there is nothing extra to install.

get_text throws away every tag and keeps only the words. separator="\n" puts each separate piece of text on its own line, and strip=True removes the spaces at the edges of each piece. The result is one long string called text.

Step 4: Saving the text to a file

    with open(file_path, "w", encoding="utf-8") as file:
        file.write(text)
    return file_path

open(file_path, "w") creates the file, or replaces it if a file of that name already exists. encoding="utf-8" is the setting that keeps letters from every language correct, including Chinese. file.write(text) puts the text into the file, and Python closes the file automatically when the indented block ends. That is what with does.

The last line, return file_path, is the part an agent cares about. The function gives back the name of the file, so the agent can pass that name to the next step.

Step 5: Calling the tool and reading the file back

result = scrape_webpage(
    "https://pongyuenlam.github.io/biography.html",
    "tool05_scraper.txt",
)
print(result)

with open(result, "r", encoding="utf-8") as file:
    contents = file.read()
    print(contents)

The first statement calls the tool with two values: the page to read and the file to write. The returned file name is stored in result, and print(result) shows it. The second block opens that same file, reads it back with "r" for reading, and prints the contents, so you can check that the page really arrived.

The real run prints this, trimmed to the first and last lines:

tool05_scraper.txt
Biography | Howard Lam, PhD
Howard Lam, PhD
Home
Biography
Research
Cases
Books
AI Notes
Links
Glossary
Biography
Howard Pong Yuen Lam (林邦源), PhD has spent over three decades turning bold ideas
    into measurable results across the boardroom and the classroom. A
    scholar, educator, and marketing strategist, he combines rigorous
    thinking, creative innovation, and hands-on leadership to lasting effect.
...
© 2026 Howard Pong Yuen LAM, PhD
Email
GitHub
Google Scholar
ORCID
LinkedIn

The first line is the file name. Everything after it is the page as plain text: 56 lines in all, with the real biography in the middle and the menu and footer links around it.

The whole program

The pieces above, put back together. This is the complete file, 31 lines in all:

# https://pypi.org/project/beautifulsoup4/
import requests
from bs4 import BeautifulSoup
def scrape_webpage(url: str, file_path: str) -> str:
    """
    Fetch a webpage and save its visible text content into a .txt file.

    Args:
        url (str): The webpage URL to scrape.
        file_path (str): Destination path for the .txt file.

    Returns:
        str: The file path that was written.
    """
    response = requests.get(url, timeout=10)
    soup = BeautifulSoup(response.text, "html.parser")
    text = soup.get_text(separator="\n", strip=True)

    with open(file_path, "w", encoding="utf-8") as file:
        file.write(text)
    return file_path

result = scrape_webpage(
    "https://pongyuenlam.github.io/biography.html",
    "tool05_scraper.txt",
)
print(result)

with open(result, "r", encoding="utf-8") as file:
    contents = file.read()
    print(contents)

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the program and a pyproject.toml file that lists the two packages it needs. Open the folder in Visual Studio Code, then type these three lines in the terminal, one at a time:

uv python pin 3.12.10
uv sync
uv run tool05_scraper.py

The first line records which Python version this folder uses. The second installs requests and beautifulsoup4 into a private environment. The third runs the program and prints the output shown above.

Those three lines are needed only once. The next time you want to run the program again, type just uv run tool05_scraper.py.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

Where it goes wrong

Further reading