如何使用 AI 代理删除草稿中的重复句子(Research,版本 5)?
上一个版本用了三个工具:
search_internet、
generate_draft 和
write_draft。
这个版本加入第四个工具
enhance_draft,它读取已完成的草稿,删去没有
带来新东西的句子。每一句仍然是原文照抄,每一个尾注仍然指向一个网页。只是
两者的数目都少了,而剩下的每一句都有它留下的理由。
每次运行只搜索一次
在新工具之前,搜索工具有一点小改动。它现在把答案存进名为
tavily_search.json 的文件,如果文件已经存在,
就直接读取它。
if os.path.exists("tavily_search.json"):
with open("tavily_search.json", "r", encoding="utf-8") as f:
response = json.load(f)
else:
response = tavily_client.search(query, max_results=10)
with open("tavily_search.json", "w", encoding="utf-8") as f:
json.dump(response, f, ensure_ascii=False, indent=2)
os.path.exists 问一个简单问题:这个文件是否
已经在文件夹里?如果已在,就用 json.load 读回
已保存的答案,不再搜索。如果不在,就调用 Tavily,并用
json.dump 把答案写出来。
在程序接近底部的位置,文件会在运行开始之前被删除,所以每次运行的第一次 搜索都是最新的:
if os.path.exists("tavily_search.json"):
os.remove("tavily_search.json") # start each run with a fresh search
indent=2 让你打开文件时看得清楚,而
ensure_ascii=False 让非英文字符保持原样,不会
变成转义码。运行完之后,这个文件值得打开看看:里面是 Tavily 返回的全部
内容,不只是屏幕上显示的 200 个字符。
另外两个共享的列表
generate_draft 工具本来就要把它的成果放在下
一个工具找得到的地方。enhance_draft 工具还需要
另外两个地方:
draft_items = []
skipped_items = []
draft_items 存放草稿中每一句,连同它来自的
网页,也就是新工具读取的内容。skipped_items
存放被删去的句子。
第四个工具:enhance_draft
return_direct=True 放在那个结果应该成为最终
答案的工具上。
@tool(return_direct=True)
def enhance_draft(keep: list[int]):
"""
Write an enhanced draft from the draft made by write_draft.
keep: the endnote numbers of the sentences to keep. Sentence 1 is always kept.
Read sentences 2, 3, ... one by one, and keep a sentence only if it adds
useful new information that the sentences already kept do not give.
"""
输入是一串整数:keep: list[int]。哪些句子
留下,由模型决定 —— 这是关于意思的判断,而模型擅长这件事。之后的代码
负责精确的部分:重新编号尾注,并记录被丢弃的句子。
工具内部
skipped_items = []
s = ""
endnotes = ""
endnote_num = 1
for i, item in enumerate(draft_items, start=1):
if i != 1 and i not in keep:
skipped_items.append({"num": i, "sentence": item["sentence"], "url": item["url"]})
continue # skip sentences that add little new information
s = s + item["sentence"] + "[" + str(endnote_num) + "]\n"
endnotes = endnotes + f"[{endnote_num}] {item['url']}\n"
endnote_num = endnote_num + 1
enumerate(draft_items, start=1) 逐项走过草稿,
每次都交出两样东西:位置 i 和该项本身。
位置从 1 开始而不是 0,因为编号必须与读者看到的尾注号码一致。
这个判断由两个条件以 and 连接。
i != 1 保护第一句:无论模型说什么,它都会
保留。然后 i not in keep 问这一句是否属于
模型选择舍弃的句子。两个条件都成立,句子才会被删去。
被删去的句子会连同原有编号、文字和链接写进
skipped_items,程序之后就可以报告它。
continue 然后跳到下一句,也就是下面两行不会
运行,这一句不会得到尾注号码。计数器只为保留下来的句子移动,所以新草稿的
编号是 [1]、
[2]、[3]
……,中间没有跳号。
在演示运行中,模型保留了第 1、2、4、6、7 句,共 5 句,所以它们在增强版 草稿中成为第 1、2、3、4、5 句。
AI 代理
agent = create_agent(
model=model,
tools=[search_internet, generate_draft, write_draft, enhance_draft],
system_prompt="""
You are a helpful assistant.
Always call search_internet, using the user's message exactly as the query.
Do not rephrase, shorten, translate, or add anything to it.
Then call generate_draft to select the useful results.
Then go through each sentence in each selected result, pick the one sentence
closest in meaning to the user's query, and copy it exactly.
A complete sentence has a subject and a verb and ends with a full stop;
table rows, headings, menus and fragments are not complete sentences.
If a result has no complete sentence, use an empty string "" for it.
Call write_draft with these sentences in the same order as the results.
Then read the draft from write_draft. Keep sentence 1.
Read the remaining sentences one by one, and keep a sentence only if it adds
useful new information that the sentences already kept do not give.
Skip a sentence that only repeats a similar figure already given.
Keep a sentence that adds a new fact, such as a ranking, a trend,
a breakdown, or an official government source.
Call enhance_draft with the endnote numbers of the sentences to keep.
""",
)
工具列表现在有
search_internet、
generate_draft、
write_draft 和
enhance_draft。提示词就是告诉模型该调用哪一个
工具、以及先后顺序的东西。
提示词中两句关键的句子如下: “逐句阅读其余的句子,只有某句带来已保留句子没有的有用新信息时,才 保留它。”这两句可以帮我们减少句子重复。
结果
以下代码块印出两个工具的结果:
write_draft 和
enhance_draft。
for message in result["messages"]:
if message.type == "tool" and message.name in ["write_draft", "enhance_draft"]:
print(f"Tool: {message.name}\n")
print(message.content)
elif message.type == "tool":
response = json.loads(message.content)
print(f"Tool: {message.name}")
以下印出被 enhance_draft 跳过的句子,供我们
参考:
print("Sentences skipped by enhance_draft:\n")
if skipped_items == []:
print("None\n")
for item in skipped_items:
print(f"[{item['num']} in write_draft] {item['sentence']}")
print(f" {item['url']}\n")
输出结果
屏幕现在印出四个工具区块 ——
search_internet、
generate_draft、
write_draft 和
enhance_draft —— 然后是被删去的句子列表。
以下是增强版草稿:
Tool: enhance_draft
China, Hong Kong SAR 2026 population is estimated at 7,378,602 people at mid-year.[1]
As of 2026, Hong Kong has a total population of 7,378,602, ranking as the 107th most populous nation in the world.[2]
Total current population for Hong Kong in 2026 is 7,500,962, a 0.02% increase from 2025.[3]
Hong Kong's population was approximately 7.50 million in mid-2025.[4]
Hong Kong is a largely homogenous society, with about 91.6% of its people being Chinese (ethnically speaking, Han Chinese).[5]
Endnotes:
[1] https://www.worldometers.info/world-population/china-hong-kong-sar-population
[2] https://worldpopulationreview.com/countries/hong-kong
[3] https://www.macrotrends.net/global-metrics/countries/hkg/hong-kong/population
[4] https://www.gov.hk/en/about/abouthk/facts.htm
[5] https://www.had.gov.hk/rru/en/info/demographics.htm
七句变成五句,尾注从 1 重新编号,所以中间没有缺号。
接着是让这个选择可以被核对的部分。这次运行删去了这些:
Sentences skipped by enhance_draft:
[3 in write_draft] The population of Hong Kong is 7.5 million.
https://study.com/academy/lesson/hong-kong-facts-population.html
[5 in write_draft] In 2023, the estimated population of Hong Kong was 7.443 million.
https://ourworldindata.org/profile/population-demography/hong-kong
两句被丢弃的原因相同:这个数字已经有了。study.com 的页面说 750 万,没有 注明年份;Our World in Data 的页面给出 2023 年的 744.3 万;而草稿已经有 2026 年的 7,500,962 和 2025 年中的 750 万。
结果并不完美,这一点值得说明。第 [1] 句和
第 [2] 句都说 7,378,602,无论怎样衡量都是
重复;第二句之所以留下,是因为它同时加上了第 107 位的排名。如果工具把所有
出现两次的内容都删去,排名也会被一起拿走。
留下的五句仍然互相矛盾:7,378,602、7,500,962 和 750 万,是同一个问题的 三个不同答案。删去重复并不能消除分歧,这个版本也没有这样声称。
我们可以再加一个工具来改进提示词,例如
generate_summary,读取每句中的数值,并说出
例如所找到数字的平均值或中位数。
完整程序
import os
import json
from dotenv import load_dotenv
from tavily import TavilyClient
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from langchain.agents import create_agent
load_dotenv()
DEEPSEEK_API_KEY = os.getenv("DEEPSEEK_API_KEY")
TAVILY_API_KEY = os.getenv("TAVILY_API_KEY")
tavily_client = TavilyClient(api_key=TAVILY_API_KEY)
last_search = {}
selected_results = []
draft_items = []
skipped_items = []
@tool
def search_internet(query: str):
"""
Search the internet for up-to-date information about the query
and return the full Tavily search response.
"""
global last_search
if os.path.exists("tavily_search.json"):
with open("tavily_search.json", "r", encoding="utf-8") as f:
response = json.load(f)
else:
response = tavily_client.search(query, max_results=10)
with open("tavily_search.json", "w", encoding="utf-8") as f:
json.dump(response, f, ensure_ascii=False, indent=2)
last_search = response
return response
@tool
def generate_draft():
"""
Go through the results from the latest search_internet call.
Keep results with a score of 0.5 or higher, but never keep Wikipedia.
Return the selected results.
"""
global selected_results
selected = []
for r in last_search.get("results", []):
if r["score"] >= 0.5 and "wikipedia.org" not in r["url"]:
selected.append(r)
selected_results = selected
return {"query": last_search.get("query"), "results": selected}
@tool
def write_draft(sentences: list[str]):
"""
Write a draft with endnotes from the results of generate_draft.
sentences: one sentence per result from generate_draft, in the same order.
Each sentence must be copied exactly from that result's content, and be
the sentence closest in meaning to the user's query.
Use an empty string "" for a result that has no complete sentence.
"""
global draft_items
draft_items = []
s = ""
endnotes = ""
endnote_num = 1
for sentence, r in zip(sentences, selected_results):
if sentence.strip() == "":
continue # skip results with no complete sentence
s = s + sentence + "[" + str(endnote_num) + "]\n"
endnotes = endnotes + f"[{endnote_num}] {r['url']}\n"
draft_items.append({"sentence": sentence, "url": r["url"]})
endnote_num = endnote_num + 1
s = s + "\nEndnotes:\n" + endnotes
return s
@tool(return_direct=True)
def enhance_draft(keep: list[int]):
"""
Write an enhanced draft from the draft made by write_draft.
keep: the endnote numbers of the sentences to keep. Sentence 1 is always kept.
Read sentences 2, 3, ... one by one, and keep a sentence only if it adds
useful new information that the sentences already kept do not give.
"""
global skipped_items
skipped_items = []
s = ""
endnotes = ""
endnote_num = 1
for i, item in enumerate(draft_items, start=1):
if i != 1 and i not in keep:
skipped_items.append({"num": i, "sentence": item["sentence"], "url": item["url"]})
continue # skip sentences that add little new information
s = s + item["sentence"] + "[" + str(endnote_num) + "]\n"
endnotes = endnotes + f"[{endnote_num}] {item['url']}\n"
endnote_num = endnote_num + 1
s = s + "\nEndnotes:\n" + endnotes
return s
model = ChatOpenAI(
model="deepseek-flash",
api_key=DEEPSEEK_API_KEY,
base_url="https://api.deepseek.com",
temperature=0,
)
agent = create_agent(
model=model,
tools=[search_internet, generate_draft, write_draft, enhance_draft],
system_prompt="""
You are a helpful assistant.
Always call search_internet, using the user's message exactly as the query.
Do not rephrase, shorten, translate, or add anything to it.
Then call generate_draft to select the useful results.
Then go through each sentence in each selected result, pick the one sentence
closest in meaning to the user's query, and copy it exactly.
A complete sentence has a subject and a verb and ends with a full stop;
table rows, headings, menus and fragments are not complete sentences.
If a result has no complete sentence, use an empty string "" for it.
Call write_draft with these sentences in the same order as the results.
Then read the draft from write_draft. Keep sentence 1.
Read the remaining sentences one by one, and keep a sentence only if it adds
useful new information that the sentences already kept do not give.
Skip a sentence that only repeats a similar figure already given.
Keep a sentence that adds a new fact, such as a ranking, a trend,
a breakdown, or an official government source.
Call enhance_draft with the endnote numbers of the sentences to keep.
""",
)
if os.path.exists("tavily_search.json"):
os.remove("tavily_search.json") # start each run with a fresh search
prompt = "What is the population of Hong Kong?"
result = agent.invoke({
"messages": [{"role": "user", "content": prompt}]
})
for message in result["messages"]:
if message.type == "tool" and message.name in ["write_draft", "enhance_draft"]:
print(f"Tool: {message.name}\n")
print(message.content)
elif message.type == "tool":
response = json.loads(message.content)
print(f"Tool: {message.name}")
print(f"Query: {response['query']}\n")
for i, r in enumerate(response["results"], start=1):
print(f"{i}. {r['title']} (score: {r['score']:.2f})")
print(f" {r['url']}")
print(f" {r['content'][:200].replace(chr(10), ' ')}...\n")
print("Sentences skipped by enhance_draft:\n")
if skipped_items == []:
print("None\n")
for item in skipped_items:
print(f"[{item['num']} in write_draft] {item['sentence']}")
print(f" {item['url']}\n")
在自己的电脑上运行
下载演示程序
(Folder-AI-Research05.zip)。zip 里有三个文件:
research05.py(完整脚本)、pyproject.toml 和
README.md。这个版本会调用语言模型,所以需要
Tavily 密钥和 DeepSeek 密钥。
- 在 Visual Studio Code 中打开这个文件夹。
-
在文件夹里创建一个名为
.env的文件,内容有两行:
Tavily 密钥可以从 Tavily 控制台获取,DeepSeek 密钥则可以从 DeepSeek 控制台获取。用量由各家提供商计费,所以运行之前,请先在那里查看当前的 收费条款。TAVILY_API_KEY=your-tavily-key DEEPSEEK_API_KEY=your-deepseek-key - 在终端里逐行运行以下三行:
uv python pin 3.12.10uv syncuv run research05.py - 你应该会依次看到四个工具区块 —— 搜索的十个结果、
generate_draft的七个结果、来自write_draft的七句草稿,然后是enhance_draft缩短后的草稿 —— 接着是 被跳过句子的列表。这次运行只需要几秒钟。
如果你还没有配置好 uv,请先看 如何安装 Visual Studio Code?,然后看 uv 笔记的 Windows 或 Mac 版本。
局限
-
重复是靠判断,不是测量。没有规则规定两句要多相似才删去
其中一句,所以再运行一次可能保留另一组句子。
temperature=0会让这种情况较少出现,但不是 不可能。 -
有些重复仍然存在。演示中的第
[1]句和第[2]句两次给出同一个数字,因为第二句同时 带有排名。工具用一种多余换来一份较好的草稿。 - 数字仍然互相矛盾。三个数字都留下来了,而草稿无法说出 哪一个正确。
延伸阅读
- 如何使用 AI 代理撰写附有尾注的草稿(Research,版本 4)? —— 七句草稿的来源。
- 如何使用 AI 代理挑选有用的搜索结果(Research,版本 3)? —— 第一个就保留哪些结果做出判断的工具。
- LangChain —— 文档 —— 工具、
return_direct和create_agent。 - Tavily —— 文档 ——
search调用及其响应字段。