How does a program split a text into sentences?
A program can take in text from many places: a text file on disk, or a webpage fetched over the internet. Either way, the program usually gets the text one line at a time. A line, however, is not a sentence: one line can hold several sentences, and a sentence can run over a line break. Many tasks, such as counting sentences or feeding text to an AI model, need the text as a list of sentences instead.
The obvious idea is to cut at every full stop. That fails at once, because a full stop does other jobs: it ends abbreviations such as e.g. and Dr., and it sits inside numbers such as 3.14. The split must therefore look at more than the full stop alone, and that is what the code in this note does with five lines:
sentences = []
for line in lines:
for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
sentences.append(sentence)
lines = sentences
The rest of this note explains these five lines in plain words, then shows them inside a small program you can run.
What a regular expression is
The long piece in quotation marks is a regular expression: a search pattern
written in a small notation of its own. Instead of looking for fixed words, a regular expression
describes a shape, such as “a full stop, then a space, then a capital
letter”. Python's re module does the matching; re is short for
regular expression.
re.split(pattern, line) cuts the line everywhere the pattern matches and hands back
the pieces as a list.
The pattern in three parts
(?<=[.!?])\s+(?=[A-Z])
The pattern says where to cut, and it has three parts.
(?<=[.!?])— look behind. The cut must come right after a., a!or a?. The square brackets list the allowed characters, and the(?<=...)wrapping means “check what is behind, but do not include it in the cut”, so the punctuation stays at the end of its sentence.\s+— the whitespace.\sstands for any whitespace character: a space, a tab, or a line break. The+means “one or more in a row”. This is the only part of the pattern that is actually removed: it becomes the cut point.(?=[A-Z])— look ahead. The character right after the whitespace must be a capital letter,AtoZ. The(?=...)wrapping again means “check but do not include”, so the capital letter stays at the start of the next sentence.
Read as one sentence, the pattern says: cut at the whitespace that sits between sentence-ending punctuation and a capital letter. That is a reasonable working definition of a sentence boundary in typed English.
The five lines around it
The first line, sentences = [], creates an empty list to collect the results. The
square brackets are how Python writes a list, and there is nothing in it yet.
The next three lines are a loop inside a loop. The outer loop goes through the text line by
line. The inner loop takes each line, splits it wherever the pattern matches, and goes through
the pieces one by one. sentences.append(sentence) adds each piece to the end of
the list.
The last line, lines = sentences, replaces the list of lines with the list of
sentences. From that point on, the name lines refers to sentences.
The whole program
The five lines above, inside a complete program. It reads a file called
my_text.txt, splits its lines into sentences, and prints one sentence per line,
12 lines in all:
import re
with open("my_text.txt", encoding="utf-8") as f:
lines = f.readlines()
sentences = []
for line in lines:
for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
sentences.append(sentence)
lines = sentences
for sentence in lines:
print(sentence.strip())
import re brings in the regular expression module. The with open(...)
line opens the file for reading; encoding="utf-8" tells Python how the characters
are stored, so any character in the file comes out correctly. UTF-8 is short for
Unicode Transformation Format: Unicode is the standard that gives every
character in every language its own number, and UTF-8 is the most common way of writing those
numbers to disk. f.readlines() reads
the whole file and gives back its lines as a list.
The last two lines print the result. Each sentence may still carry an invisible line-break
character at its end, and strip() removes it so every sentence prints on its own
line.
Running it on your own computer
Download the zip file and unzip
it to a folder. The zip holds the program, a sample my_text.txt, and a
pyproject.toml project file. Open the folder in Visual Studio Code, then type
these three lines in the terminal, one at a time:
uv python pin 3.12.10
uv sync
uv run split_sentences.py
The first line records which Python version this folder uses. The second line sets up the project; this program needs no extra packages, so there is nothing to install. The third line runs the program. With the sample file, the output is:
The rain stopped just before noon.
Did the football match go ahead?
Yes, it did!
The crowd cheered as the players returned.
By three o'clock the sun was out.
The sample file holds two lines and five sentences, which shows the point: sentences and lines
are not the same thing. Edit my_text.txt and put in your own text to try it.
If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.
Limitations
The pattern is a working definition, not a perfect one, and seeing where it fails is the best way to remember what it really says.
- Abbreviations with capitals. “Dr. Lam arrived.” splits into “Dr.” and “Lam arrived.”, because the space after Dr. is followed by a capital. The pattern cannot tell a title from a sentence end.
- Numbers. “It costs 3.14 dollars.” does not split, but only by luck: there is no whitespace after the point in 3.14, so the pattern never considers it.
- Lowercase after punctuation. “…and then it rained. nobody moved.” does not split, because a small letter fails the capital-letter check, and the two sentences stay joined.
For a text that is typed in the usual way, with standard punctuation and capitalisation, the five lines do well. For messier text, a larger tool such as a sentence-splitting library does the same job with many more rules, but the idea stays the one you have seen here.
Further reading
- Python's re module documentation lists all the shorthand codes such as
\s. - regex101.com lets you paste a pattern and a text and watch the matches light up.
- How to mitigate AI hallucination with RAG by providing a webpage? shows the same idea one step later: splitting webpage text into chunks before an AI model reads it.