How does a program split a text into sentences (ReAct, version 8)?
Version 9 of our agent will read web pages. A web page arrives as big blocks of text — headings, paragraphs, list items — and the agent works better when the text is cut into sentences first. One line of Python does the cutting. This note walks through that line, step by step, using three blocks we made up, so there is no web page in the way.
The one line to learn
sentences.extend(re.split(r"(?<=[.!?])\s+", block))
This line does two jobs at once. re.split(...) cuts one block
of text into sentences, and sentences.extend(...) adds those
sentences, one by one, to a list. The cutting rule is the part worth
understanding, and it is easiest to see by trying two simpler rules
first. Here are the three blocks we will cut:
blocks = [
"Howard Lam, PhD", # a heading: no full stop at all
"I teach AI for marketing. Do you like it? Yes!", # a paragraph: three sentences
"My average meal cost is HK$60.10 in 2025.", # 60.10 has a "." but no space after it
]
Three blocks, chosen on purpose. The heading has no full stop at all. The paragraph holds three sentences. And the price HK$60.10 contains a dot that is not the end of a sentence.
Step 1: cut on spaces — too many pieces
re.split(r"\s+", blocks[1])
['I', 'teach', 'AI', 'for', 'marketing.', 'Do', 'you', 'like', 'it?', 'Yes!']
\s means one space (or tab, or line break), and
\s+ means one or more of them in a row. Cutting at every
space gives every word its own piece — far too many. We wanted three
sentences, not ten pieces.
Step 2: cut on the punctuation — the punctuation is lost
re.split(r"[.!?]\s+", blocks[1])
['I teach AI for marketing', 'Do you like it', 'Yes!']
[.!?] means one character that is a .,
! or ?, and the \s+ after it means
the cut only happens when the punctuation is followed by a space.
'I teach AI for marketing.' becomes
'I teach AI for marketing' without the .
because there is space after the .
'Do you like it?' becomes
'Do you like it' because there is space after
?
'Yes!' becomes 'Yes!' because there is
no space after ! to cut.
Step 3: the real line — keep the punctuation
re.split(r"(?<=[.!?])\s+", blocks[1])
['I teach AI for marketing.', 'Do you like it?', 'Yes!']
(?<=[.!?]) is called a look-behind: cut only at
spaces that come right after a ., ! or
?. It looks at the punctuation but does not cut it out, so
each sentence keeps its ending. Three sentences, punctuation intact.
Step 4: each block on its own — why HK$60.10 is not cut
['Howard Lam, PhD']
['I teach AI for marketing.', 'Do you like it?', 'Yes!']
['My average meal cost is HK$60.10 in 2025.']
The heading comes back as one piece — there is no .,
! or ?, so there is nothing to cut at. And
HK$60.10 is not cut either: the rule cuts at a dot followed by a space,
and the dot in 60.10 has no space after it. The price stays inside its
sentence.
Step 5: collect every sentence into one numbered list
sentences = []
for block in blocks:
sentences.extend(re.split(r"(?<=[.!?])\s+", block))
1. Howard Lam, PhD
2. I teach AI for marketing.
3. Do you like it?
4. Yes!
5. My average meal cost is HK$60.10 in 2025.
This is what extend is for: each sentence from each block
becomes its own item in one list. Version 9 will number the sentences of
a web page exactly like this, so the agent can refer to sentence 3 or
sentence 17 by number.
The whole program
# Learn ONE line from ReAct09.py, step by step:
# sentences.extend(re.split(r"(?<=[.!?])\s+", block))
# It cuts a block of text into sentences.
# Here we make our own blocks (no web page), so we can see exactly what happens.
import re
# A few blocks, like the headings, paragraphs and list items ReAct09.py takes from a web page
blocks = [
"Howard Lam, PhD", # a heading: no full stop at all
"I teach AI for marketing. Do you like it? Yes!", # a paragraph: three sentences
"My average meal cost is HK$60.10 in 2025.", # 60.10 has a "." but no space after it
]
# ---- Step 1: split on spaces only -> cuts after EVERY word (too many pieces) ----
print("=== Step 1: re.split(r'\\s+', ...) ===")
print(re.split(r"\s+", blocks[1]))
# \s means one space (or tab, or line break); \s+ means one or more of them in a row
# ---- Step 2: split on ". ! ?" plus spaces -> the punctuation is LOST ----
print("\n=== Step 2: re.split(r'[.!?]\\s+', ...) ===")
print(re.split(r"[.!?]\s+", blocks[1]))
# [.!?] means ONE character that is . or ! or ?
# The cut happens ON the punctuation, so it disappears
# ---- Step 3: the real line -> keep the punctuation ----
print("\n=== Step 3: re.split(r'(?<=[.!?])\\s+', ...) ===")
print(re.split(r"(?<=[.!?])\s+", blocks[1]))
# (?<=[.!?]) is a "look-behind": cut only at spaces that come right AFTER . ! or ?
# It only LOOKS at the punctuation and does not cut it out, so the sentence keeps its "."
# ---- Step 4: every block -> why there is no cut inside HK$60.10 ----
print("\n=== Step 4: each block on its own ===")
for block in blocks:
print(re.split(r"(?<=[.!?])\s+", block))
# "Howard Lam, PhD" has no . ! ? so we get the whole block back as ONE sentence
# "HK$60.10" has a "." but no space after it, so it is not cut: the block stays ONE sentence
# ---- Step 5: put the sentences of ALL blocks into one list, and number them (as ReAct09.py does) ----
sentences = []
for block in blocks:
sentences.extend(re.split(r"(?<=[.!?])\s+", block)) # extend adds each sentence as its own item
print(f"\n=== Step 5: {len(sentences)} sentences ===")
for number, sentence in enumerate(sentences, start=1):
print(f"{number}. {sentence}")
# Try it: change a block to "Dr. Lam teaches AI." and run again.
# Why does it cut after "Dr."? (Hint: the rule only sees ". " — it does not know "Dr." is not a sentence end.)
Running it on your own computer
Download the demo program
(Folder-AI-ReAct08.zip). The zip holds three files:
ReAct08.py (the full script), pyproject.toml and
README.md. There is nothing to install —
re comes with Python — and no API key is needed.
- Open the folder in Visual Studio Code.
- In the Terminal, run these three lines, one at a time:
uv python pin 3.12.10uv syncuv run ReAct08.py - You should see the five steps, ending with the numbered list:
=== Step 1: re.split(r'\s+', ...) === ['I', 'teach', 'AI', 'for', 'marketing.', 'Do', 'you', 'like', 'it?', 'Yes!'] === Step 2: re.split(r'[.!?]\s+', ...) === ['I teach AI for marketing', 'Do you like it', 'Yes!'] === Step 3: re.split(r'(?<=[.!?])\s+', ...) === ['I teach AI for marketing.', 'Do you like it?', 'Yes!'] === Step 4: each block on its own === ['Howard Lam, PhD'] ['I teach AI for marketing.', 'Do you like it?', 'Yes!'] ['My average meal cost is HK$60.10 in 2025.'] === Step 5: 5 sentences === 1. Howard Lam, PhD 2. I teach AI for marketing. 3. Do you like it? 4. Yes! 5. My average meal cost is HK$60.10 in 2025.
If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.
Limitations
-
The rule sees a dot and a space, nothing more. Change a
block to
"Dr. Lam teaches AI."and run again: it cuts after"Dr.", because the rule cannot know that"Dr."is a title, not the end of a sentence. - It only works for languages that use spaces. In languages such as Chinese, there is no space after the punctuation, so this rule has nothing to cut at and a different rule is needed.
Further reading
- Python re module documentation —
split, the special sequences like\s, and look-behinds. - regex101 — paste the pattern and the blocks, and watch where it cuts.
- How to use LangChain in an AI agent (ReAct, version 7)? — the previous note in this series.