How does a program split a text into sentences?

Howard Pong Yuen Lam · AI Notes · November 3, 2026 · 4 min read

English 繁體中文 简体中文

A program can take in text from many places: a text file on disk, or a webpage fetched over the internet. Either way, the program usually gets the text one line at a time. A line, however, is not a sentence: one line can hold several sentences, and a sentence can run over a line break. Many tasks, such as counting sentences or feeding text to an AI model, need the text as a list of sentences instead.

The obvious idea is to cut at every full stop. That fails at once, because a full stop does other jobs: it ends abbreviations such as e.g. and Dr., and it sits inside numbers such as 3.14. The split must therefore look at more than the full stop alone, and that is what the code in this note does with five lines:

sentences = []
for line in lines:
    for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
        sentences.append(sentence)
lines = sentences

The rest of this note explains these five lines in plain words, then shows them inside a small program you can run.

What a regular expression is

The long piece in quotation marks is a regular expression: a search pattern written in a small notation of its own. Instead of looking for fixed words, a regular expression describes a shape, such as “a full stop, then a space, then a capital letter”. Python's re module does the matching; re is short for regular expression.

re.split(pattern, line) cuts the line everywhere the pattern matches and hands back the pieces as a list.

The pattern in three parts

(?<=[.!?])\s+(?=[A-Z])

The pattern says where to cut, and it has three parts.

Read as one sentence, the pattern says: cut at the whitespace that sits between sentence-ending punctuation and a capital letter. That is a reasonable working definition of a sentence boundary in typed English.

The five lines around it

The first line, sentences = [], creates an empty list to collect the results. The square brackets are how Python writes a list, and there is nothing in it yet.

The next three lines are a loop inside a loop. The outer loop goes through the text line by line. The inner loop takes each line, splits it wherever the pattern matches, and goes through the pieces one by one. sentences.append(sentence) adds each piece to the end of the list.

The last line, lines = sentences, replaces the list of lines with the list of sentences. From that point on, the name lines refers to sentences.

The whole program

The five lines above, inside a complete program. It reads a file called my_text.txt, splits its lines into sentences, and prints one sentence per line, 12 lines in all:

import re

with open("my_text.txt", encoding="utf-8") as f:
    lines = f.readlines()

sentences = []
for line in lines:
    for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
        sentences.append(sentence)
lines = sentences

for sentence in lines:
    print(sentence.strip())

import re brings in the regular expression module. The with open(...) line opens the file for reading; encoding="utf-8" tells Python how the characters are stored, so any character in the file comes out correctly. UTF-8 is short for Unicode Transformation Format: Unicode is the standard that gives every character in every language its own number, and UTF-8 is the most common way of writing those numbers to disk. f.readlines() reads the whole file and gives back its lines as a list.

The last two lines print the result. Each sentence may still carry an invisible line-break character at its end, and strip() removes it so every sentence prints on its own line.

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the program, a sample my_text.txt, and a pyproject.toml project file. Open the folder in Visual Studio Code, then type these three lines in the terminal, one at a time:

uv python pin 3.12.10
uv sync
uv run split_sentences.py

The first line records which Python version this folder uses. The second line sets up the project; this program needs no extra packages, so there is nothing to install. The third line runs the program. With the sample file, the output is:

The rain stopped just before noon.
Did the football match go ahead?
Yes, it did!
The crowd cheered as the players returned.
By three o'clock the sun was out.

The sample file holds two lines and five sentences, which shows the point: sentences and lines are not the same thing. Edit my_text.txt and put in your own text to try it.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

Limitations

The pattern is a working definition, not a perfect one, and seeing where it fails is the best way to remember what it really says.

For a text that is typed in the usual way, with standard punctuation and capitalisation, the five lines do well. For messier text, a larger tool such as a sentence-splitting library does the same job with many more rules, but the idea stays the one you have seen here.

Further reading