How to create a web scraping tool for an AI agent?
An AI agent can plan, choose a function and use the answer. It cannot open a webpage by itself. If you ask about a page it has never read, it either says it does not know or invents something that sounds right.
A tool is how you close that gap: an ordinary Python function the agent is allowed to call. This note builds the scraping tool from the agent toolkit. It downloads a page, keeps the readable text, saves that text to a file, and returns the name of the file. The whole tool is 31 lines.
Step 1: The function and its description
def scrape_webpage(url: str, file_path: str) -> str:
"""
Fetch a webpage and save its visible text content into a .txt file.
Args:
url (str): The webpage URL to scrape.
file_path (str): Destination path for the .txt file.
Returns:
str: The file path that was written.
"""
The first line names the function, scrape_webpage, and lists what it needs: a
webpage address in url and a destination in file_path. The
-> str at the end says the function gives back text, in this case the name of the
file it wrote.
The text between the triple quotation marks is a docstring. It is not a comment. When this function is offered to an agent, the docstring becomes the description the model reads before deciding whether to call it. It says what the tool does, what the two inputs are, and what comes back, which is exactly what a model needs in order to use the tool correctly.
Step 2: Downloading the page
response = requests.get(url, timeout=10)
The requests package downloads pages from the internet. requests.get
asks the website for the page at url, and response holds the answer.
timeout=10 gives up after ten seconds instead of waiting forever. If the address is
wrong or the site refuses, requests stops the program with an error rather than
returning an empty page.
Step 3: Keeping only the readable text
soup = BeautifulSoup(response.text, "html.parser")
text = soup.get_text(separator="\n", strip=True)
What came back is HTML, the code that describes a page. BeautifulSoup, from the
bs4 package, reads that code into a structure the program can inspect; the result is
named soup. "html.parser" is the parser built into Python, so there is
nothing extra to install.
get_text throws away every tag and keeps only the words. separator="\n"
puts each separate piece of text on its own line, and strip=True removes the spaces
at the edges of each piece. The result is one long string called text.
Step 4: Saving the text to a file
with open(file_path, "w", encoding="utf-8") as file:
file.write(text)
return file_path
open(file_path, "w") creates the file, or replaces it if a file of that name already
exists. encoding="utf-8" is the setting that keeps letters from every language
correct, including Chinese. file.write(text) puts the text into the file, and Python
closes the file automatically when the indented block ends. That is what with does.
The last line, return file_path, is the part an agent cares about. The function gives
back the name of the file, so the agent can pass that name to the next step.
Step 5: Calling the tool and reading the file back
result = scrape_webpage(
"https://pongyuenlam.github.io/biography.html",
"tool05_scraper.txt",
)
print(result)
with open(result, "r", encoding="utf-8") as file:
contents = file.read()
print(contents)
The first statement calls the tool with two values: the page to read and the file to write. The
returned file name is stored in result, and print(result) shows it. The
second block opens that same file, reads it back with "r" for reading, and prints
the contents, so you can check that the page really arrived.
The real run prints this, trimmed to the first and last lines:
tool05_scraper.txt
Biography | Howard Lam, PhD
Howard Lam, PhD
Home
Biography
Research
Cases
Books
AI Notes
Links
Glossary
Biography
Howard Pong Yuen Lam (林邦源), PhD has spent over three decades turning bold ideas
into measurable results across the boardroom and the classroom. A
scholar, educator, and marketing strategist, he combines rigorous
thinking, creative innovation, and hands-on leadership to lasting effect.
...
© 2026 Howard Pong Yuen LAM, PhD
Email
GitHub
Google Scholar
ORCID
LinkedIn
The first line is the file name. Everything after it is the page as plain text: 56 lines in all, with the real biography in the middle and the menu and footer links around it.
The whole program
The pieces above, put back together. This is the complete file, 31 lines in all:
# https://pypi.org/project/beautifulsoup4/
import requests
from bs4 import BeautifulSoup
def scrape_webpage(url: str, file_path: str) -> str:
"""
Fetch a webpage and save its visible text content into a .txt file.
Args:
url (str): The webpage URL to scrape.
file_path (str): Destination path for the .txt file.
Returns:
str: The file path that was written.
"""
response = requests.get(url, timeout=10)
soup = BeautifulSoup(response.text, "html.parser")
text = soup.get_text(separator="\n", strip=True)
with open(file_path, "w", encoding="utf-8") as file:
file.write(text)
return file_path
result = scrape_webpage(
"https://pongyuenlam.github.io/biography.html",
"tool05_scraper.txt",
)
print(result)
with open(result, "r", encoding="utf-8") as file:
contents = file.read()
print(contents)
Running it on your own computer
Download the zip file and unzip it to
a folder. The zip holds the program and a pyproject.toml file that lists the two
packages it needs. Open the folder in Visual Studio Code, then type these three lines in the
terminal, one at a time:
uv python pin 3.12.10
uv sync
uv run tool05_scraper.py
The first line records which Python version this folder uses. The second installs
requests and beautifulsoup4 into a private environment. The third runs
the program and prints the output shown above.
Those three lines are needed only once. The next time you want to run the program again, type
just uv run tool05_scraper.py.
If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.
Where it goes wrong
- Some sites refuse the request. This tool sends no
User-Agentheader, which is the short name that tells a website who is asking. Many sites reject requests that do not look like a browser. The scraper in How does web scraping turn a webpage into a file of sentences? sends one for that reason. - Pages built by JavaScript arrive nearly empty. The download only sees the HTML the server sends. If the article appears after the page loads, the text file will not contain it.
- The file is not clean.
get_textkeeps everything that is not a tag: menus, cookie notices, footer links. The run above is a good example, with the menu at the top and the LinkedIn link at the bottom. - Chinese and other non-Latin pages can look wrong. Without naming an encoding
for the download, the program relies on what the site declares. Note 13 sets
apparent_encodingto handle pages that declare it incorrectly. - Only scrape what you are allowed to read. A site's
robots.txtfile and its terms of use say what may be fetched. This tool cannot tell the difference, so the rule applies to whoever runs it.
Further reading
- How does web scraping turn a webpage into a file of sentences? — the same idea taken one step further, with sentence splitting and a cleaner text extraction.
- How to create a time tool for an AI agent? — the simplest tool of all, and why a docstring matters.
- How to create a weather tool for an AI agent? — a tool that calls a free online service instead of a webpage.
- Beautiful Soup documentation —
get_text()and the other ways to search a page. - requests documentation — headers, timeouts and error handling.