English 繁體中文 简体中文

程序如何把一段文字分成句子(ReAct,版本 8)?

林邦源博士 · AI 笔记 · 2026 年 11 月 9 日 · 约 3 分钟阅读

我们代理的版本 9 将会读取网页。网页送来的时候是一大块一大块的文字 —— 标题、段落、列表项 —— 如果先把文字切成句子,代理的表现会更好。真正 负责切分的只有一行 Python。这篇笔记就用我们自己编的三个文字块,逐步 拆解这一行,过程中不涉及任何网页。

要学的那一行

sentences.extend(re.split(r"(?<=[.!?])\s+", block))

这一行同时做两件事。re.split(...) 把一个文字块切成句子, sentences.extend(...) 则把这些句子逐一加入一个列表。真正 值得理解的是切分的规则,而最容易的方法,是先试两个更简单的规则。以下是 我们要切开的三个块:

blocks = [
    "Howard Lam, PhD",                                    # a heading: no full stop at all
    "I teach AI for marketing. Do you like it? Yes!",     # a paragraph: three sentences
    "My average meal cost is HK$60.10 in 2025.",          # 60.10 has a "." but no space after it
]

三个块都是刻意挑选的。标题完全没有句号。段落里有三个句子。而价格 HK$60.10 中间的那一点,并不是句子的结尾。

第 1 步:按空格切 —— 碎片太多

re.split(r"\s+", blocks[1])
['I', 'teach', 'AI', 'for', 'marketing.', 'Do', 'you', 'like', 'it?', 'Yes!']

\s 代表一个空格(或制表符,或换行),\s+ 代表连续一个或多个。在每个空格处切开,每个词都会变成独立一片 —— 实在太多。我们要的是三个句子,不是十片碎片。

第 2 步:按标点切 —— 标点不见了

re.split(r"[.!?]\s+", blocks[1])
['I teach AI for marketing', 'Do you like it', 'Yes!']

[.!?] 代表一个 .、! 或 ? 字符,紧随其后的 \s+ 表示只有在标点后面有空格 时才切开。
'I teach AI for marketing.'变成 'I teach AI for marketing',少了那个 .,因为 . 后面有空格。
'Do you like it?'变成 'Do you like it',因为 ? 后面 有空格。
'Yes!'变成 'Yes!', 因为 ! 后面没有空格可以切。

第 3 步:真正的那一行 —— 保留标点

re.split(r"(?<=[.!?])\s+", blocks[1])
['I teach AI for marketing.', 'Do you like it?', 'Yes!']

(?<=[.!?]) 称为后顾断言:只在紧接 .、 ! 或 ? 之后的空格处切开。它会查看标点,但不会 把标点切走,所以每个句子都保留自己的结尾。三个句子,标点完整。

第 4 步:逐个块处理 —— 为什么 HK$60.10 不会被切开

['Howard Lam, PhD']
['I teach AI for marketing.', 'Do you like it?', 'Yes!']
['My average meal cost is HK$60.10 in 2025.']

标题原封不动地回来 —— 里面没有 .、! 或 ?,所以没有可以切开的位置。HK$60.10 也没有被切开:规则在 “一点加一个空格”处切,而 60.10 中的那一点后面没有空格。价格就留在原来 的句子里面。

第 5 步:把所有句子收集到一个编号列表

sentences = []
for block in blocks:
    sentences.extend(re.split(r"(?<=[.!?])\s+", block))
1. Howard Lam, PhD
2. I teach AI for marketing.
3. Do you like it?
4. Yes!
5. My average meal cost is HK$60.10 in 2025.

这就是 extend 的用途:每个块的每个句子,都成为同一个列表中 的一个条目。版本 9 会用完全相同的方式为网页的句子编号,这样代理就可以用 编号指代第 3 句或第 17 句。

完整程序

# Learn ONE line from ReAct09.py, step by step:
#     sentences.extend(re.split(r"(?<=[.!?])\s+", block))
# It cuts a block of text into sentences.
# Here we make our own blocks (no web page), so we can see exactly what happens.
import re

# A few blocks, like the headings, paragraphs and list items ReAct09.py takes from a web page
blocks = [
    "Howard Lam, PhD",                                    # a heading: no full stop at all
    "I teach AI for marketing. Do you like it? Yes!",     # a paragraph: three sentences
    "My average meal cost is HK$60.10 in 2025.",          # 60.10 has a "." but no space after it
]

# ---- Step 1: split on spaces only -> cuts after EVERY word (too many pieces) ----
print("=== Step 1: re.split(r'\\s+', ...) ===")
print(re.split(r"\s+", blocks[1]))
# \s means one space (or tab, or line break); \s+ means one or more of them in a row

# ---- Step 2: split on ". ! ?" plus spaces -> the punctuation is LOST ----
print("\n=== Step 2: re.split(r'[.!?]\\s+', ...) ===")
print(re.split(r"[.!?]\s+", blocks[1]))
# [.!?] means ONE character that is . or ! or ?
# The cut happens ON the punctuation, so it disappears

# ---- Step 3: the real line -> keep the punctuation ----
print("\n=== Step 3: re.split(r'(?<=[.!?])\\s+', ...) ===")
print(re.split(r"(?<=[.!?])\s+", blocks[1]))
# (?<=[.!?]) is a "look-behind": cut only at spaces that come right AFTER . ! or ?
# It only LOOKS at the punctuation and does not cut it out, so the sentence keeps its "."

# ---- Step 4: every block -> why there is no cut inside HK$60.10 ----
print("\n=== Step 4: each block on its own ===")
for block in blocks:
    print(re.split(r"(?<=[.!?])\s+", block))
# "Howard Lam, PhD" has no . ! ? so we get the whole block back as ONE sentence
# "HK$60.10" has a "." but no space after it, so it is not cut: the block stays ONE sentence

# ---- Step 5: put the sentences of ALL blocks into one list, and number them (as ReAct09.py does) ----
sentences = []
for block in blocks:
    sentences.extend(re.split(r"(?<=[.!?])\s+", block))   # extend adds each sentence as its own item
print(f"\n=== Step 5: {len(sentences)} sentences ===")
for number, sentence in enumerate(sentences, start=1):
    print(f"{number}. {sentence}")

# Try it: change a block to "Dr. Lam teaches AI." and run again.
# Why does it cut after "Dr."? (Hint: the rule only sees ". " — it does not know "Dr." is not a sentence end.)

在自己的电脑上运行

下载示范程序 (Folder-AI-ReAct08.zip)。zip 内有三个文件: ReAct08.py(完整程序)、pyproject.toml 和 README.md。无需安装任何东西 —— re 是 Python 自带的 —— 也不需要 API 密钥。

  1. 在 Visual Studio Code 打开这个文件夹。
  2. 在终端逐行运行以下三条命令:
    uv python pin 3.12.10
    uv sync
    uv run ReAct08.py
  3. 你应该会看到五个步骤,最后是编号列表:
    === Step 1: re.split(r'\s+', ...) ===
    ['I', 'teach', 'AI', 'for', 'marketing.', 'Do', 'you', 'like', 'it?', 'Yes!']
    
    === Step 2: re.split(r'[.!?]\s+', ...) ===
    ['I teach AI for marketing', 'Do you like it', 'Yes!']
    
    === Step 3: re.split(r'(?<=[.!?])\s+', ...) ===
    ['I teach AI for marketing.', 'Do you like it?', 'Yes!']
    
    === Step 4: each block on its own ===
    ['Howard Lam, PhD']
    ['I teach AI for marketing.', 'Do you like it?', 'Yes!']
    ['My average meal cost is HK$60.10 in 2025.']
    
    === Step 5: 5 sentences ===
    1. Howard Lam, PhD
    2. I teach AI for marketing.
    3. Do you like it?
    4. Yes!
    5. My average meal cost is HK$60.10 in 2025.

如果你还没有设置好 uv,请先看如何安装 Visual Studio Code?, 然后看 Windows 或 Mac 的 uv 笔记。

限制

延伸阅读