How does an AI agent read a web page (ReAct, version 9)?
Version 9 of our agent does something new: it reads a real web page. A page is not text yet. It arrives as HTML, full of tags that tell a browser how to display things, and an agent cannot use that. Two small tools clean it up — one downloads the page, the other finds the parts that hold sentences. Then the sentence rule from version 8 finishes the job.
Two tools, one job each
import re
import requests
from bs4 import BeautifulSoup
requests is the downloader: it asks a website for a page and
hands back what the server sends. BeautifulSoup, from the
bs4 package, is the reader: it turns that HTML into something
a program can search, so we can say "give me every heading and paragraph"
instead of hunting through tags by hand.
Step 1: download the page
url = "https://pongyuenlam.github.io/"
USER_AGENT = "ReAct09-class-demo" # tells the website who is visiting
response = requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=10)
soup = BeautifulSoup(response.text, "html.parser")
requests.get asks the server for the page at url.
timeout=10 means "give up after ten seconds" so the program
never hangs. The User-Agent is a short name that travels with
the request, so the website can see who is visiting rather than receiving
an anonymous hit. What comes back sits in response.text: the
raw HTML. The last line pours that HTML into soup, ready to be
searched.
Step 2: keep only the parts that hold sentences
blocks = []
for element in soup.find_all(["h1", "h2", "h3", "p", "li"]):
text = element.get_text(" ", strip=True) # keeps the text, drops the tags
text = re.sub(r"\s+", " ", text) # many spaces become one
if text:
blocks.append(text)
find_all takes a list of tag names and returns every element
with any of those names: the headings, the paragraphs and the list items.
That is where a page keeps its sentences. get_text(" ", strip=True)
throws away the tags and keeps the words, joining the pieces with a space
and trimming the edges. re.sub(r"\s+", " ", text) then turns
runs of spaces and line breaks into a single space, because HTML is full of
them. The last two lines check that a block actually contains text, so
empty blocks are skipped.
Step 3: cut each block into sentences
sentences = []
for block in blocks:
sentences.extend(re.split(r"(?<=[.!?])\s+", block))
This is the line from version 8, unchanged. Each block is cut at spaces
that sit right after a ., ! or ?,
and the punctuation stays with its sentence. extend adds every
sentence to one list, so the whole page ends up in a single list of
sentences. Version 8 explains that line step
by step.
Step 4: print each sentence with a number
print(f"=== {len(sentences)} sentences from {url} ===")
for number, sentence in enumerate(sentences, start=1):
print(f"{number}. {sentence}")
The first line says how many sentences were found and which page they came
from. enumerate(..., start=1) counts the sentences as it walks
through the list, starting at 1. Numbering matters: later, the agent can
point at "sentence 17" instead of copying the whole text.
What comes out
=== 24 sentences from https://pongyuenlam.github.io/ ===
1. Home
2. Biography
3. Research
4. Cases
5. Books
6. AI Notes
7. Links
8. Glossary
9. Howard Lam, PhD
10. AI Marketing Researcher, Educator, Developer
11. A marketing PhD who spent two decades in brand management at Coca-Cola, McDonald's, and P&G before conducting case research, writing, and teaching at The Chinese University of Hong Kong and City University of Hong Kong, I now research how AI is reshaping marketing.
12. This site links to my work: peer-reviewed research (including the Journal of Marketing), business cases published through Harvard Business Impact and Ivey Publishing, open-source AI tools and code, and curated resources on AI and machine learning.
13. Links
...
19. © 2026 Howard Pong Yuen LAM,PhD
20. Email
21. GitHub
22. Google Scholar
23. ORCID
24. LinkedIn
Look at the first eight sentences: Home, Biography, Research, Cases, Books, AI Notes, Links, Glossary. That is the menu of the site. A menu is a list, and Step 2 was told to keep list items, so the menu arrives as sentences too. The long sentence 11 is the biography paragraph, kept whole because its commas are not sentence endings. The last sentences are the footer links.
The whole program
# Read a web page with requests + BeautifulSoup and print each sentence.
# No AI model here: this is the "reading" step an agent needs before it can use a web page.
import re
import requests
from bs4 import BeautifulSoup
url = "https://pongyuenlam.github.io/"
USER_AGENT = "ReAct09-class-demo" # tells the website who is visiting
# 1) Download the page, then let BeautifulSoup turn the HTML into something we can search
response = requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=10)
soup = BeautifulSoup(response.text, "html.parser")
# 2) Take the text of each block: headings, paragraphs and list items
blocks = []
for element in soup.find_all(["h1", "h2", "h3", "p", "li"]):
text = element.get_text(" ", strip=True) # " " joins the text of links inside the block
text = re.sub(r"\s+", " ", text) # turn line breaks and extra spaces into one space
if text:
blocks.append(text)
# 3) Split each block into sentences: after . ! or ? followed by a space
sentences = []
for block in blocks:
sentences.extend(re.split(r"(?<=[.!?])\s+", block))
# 4) Print each sentence with a number
print(f"=== {len(sentences)} sentences from {url} ===")
for number, sentence in enumerate(sentences, start=1):
print(f"{number}. {sentence}")
Running it on your own computer
Download the demo program
(Folder-AI-ReAct09.zip). The zip holds three files:
ReAct09.py (the full script), pyproject.toml and
README.md. Installing needs an internet connection, because
the program uses two outside packages, but no API key is needed.
- Open the folder in Visual Studio Code.
- In the Terminal, run these three lines, one at a time:
uv python pin 3.12.10uv syncuv run ReAct09.py - You should see the sentence count, then one numbered sentence per line.
If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.
Limitations
- It reads one page and stops. The program does not follow links, so a page that only points to other pages gives the agent very little to work with.
-
Some pages arrive almost empty. If a page is built by
JavaScript in the browser, the text is not in the HTML that
requestsdownloads, and the program finds nothing. - Menus and footers count as sentences. Anything inside a list item or paragraph is kept, whether it is real prose or a page label.
-
Be a polite visitor. Keep the
User-Agentheader so the website knows who is calling, and read pages that invite reading rather than pages that forbid it.
Further reading
- Requests documentation — how to download a page, set headers and timeouts.
- BeautifulSoup documentation —
find_all,get_textand the other ways to search HTML. - How does a program split a text into sentences (ReAct, version 8)? — the sentence rule this note reuses.