How does web scraping turn a webpage into a file of sentences?
A webpage is not plain text. Behind what you see, the page is a long piece of HTML code (HyperText Markup Language): the readable article is mixed together with the menu, the footer, the styling, and the scripts that make the page interactive. Before a program can do anything useful with a webpage — count its sentences, search it, or feed it to an AI model — it first has to pull the readable text out of that code.
Downloading a page and extracting its text with a program is called web scraping:
the program scrapes the useful content off the page, leaving the code behind. The
program in this note is a small web scraper that works in four steps: it downloads a webpage,
keeps only the parts worth reading, cuts the text into sentences, and saves the sentences to a
text file, one sentence per line. The example page is Panadol's page about headaches, and the
output file is called panadol.txt.
Step 1: Downloading the page
url = "https://www.panadol.com/en-gb/adult-symptoms/headache/"
headers = {"User-Agent": "Mozilla/5.0 (compatible; SkillBot/1.0)"}
resp = requests.get(url, headers=headers, timeout=15)
resp.raise_for_status()
resp.encoding = resp.apparent_encoding
The requests library downloads pages from the internet. The first two lines set up
the call: url is the web address, and headers carries a
User-Agent, a short name that tells the website who is asking. Many websites refuse
requests that do not look like they came from a browser, so the program introduces itself with a
browser-like name.
requests.get(...) downloads the page; resp is short for
response, the server's answer, and timeout=15 gives up after 15 seconds
instead of waiting forever. raise_for_status() stops the program with a clear error
if the server reported a failure, such as “page not found”, instead of quietly
saving an error page as if it were the article.
The last line is about encoding: the rule that turns the page's raw bytes into
readable characters. Web pages often declare their encoding wrongly, so the program switches to
apparent_encoding, the encoding requests guessed from the actual
bytes. This keeps special characters correct when the text is read.
Step 2: Keeping only the readable text
soup = BeautifulSoup(resp.text, "html.parser")
for tag in soup(["script", "style", "noscript", "nav", "footer"]):
tag.decompose()
lines = [" ".join(tag.get_text(" ").split()) for tag in soup.find_all(["h1", "h2", "h3", "p", "li"])]
lines = [line for line in lines if line != ""]
BeautifulSoup, from the bs4 package (Beautiful Soup 4), parses the
HTML code into a tree of tags that the program can search; the result is named
soup. The next two lines throw away five kinds of tag we do not want:
script (JavaScript code), style (page styling), noscript
(text shown when JavaScript is off), nav (the menu) and footer (the
bottom of the page). decompose() deletes each tag and everything inside it, so only
the real content is left.
Then find_all collects the tags worth reading: headings (h1 is the
largest, h3 smaller), paragraphs (p) and list items
(li). For each tag, get_text(" ") takes out just the words, and
" ".join(... .split()) squeezes any run of spaces and line breaks into single
spaces, so each tag becomes one tidy line of text. The second line drops empty lines, keeping
only lines that really contain words.
Step 3: Cutting the lines into sentences
sentences = []
for line in lines:
for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
sentences.append(sentence)
lines = sentences
One line can hold several sentences, so the program cuts each line wherever a space comes after
a full stop, exclamation mark or question mark and is followed by a capital letter. The pattern
is a regular expression, a search pattern handled by Python's re
module, and the five lines around it are explained piece by piece in
How does a program split a text into sentences? After these lines,
the name lines refers to sentences, not webpage lines.
Step 4: Saving the sentences to a file
with open("panadol.txt", "w", encoding="utf-8") as outfile:
for line in lines:
outfile.write(line + "\n")
open("panadol.txt", "w", ...) opens the file in the same folder for writing,
creating it if it is not there and emptying it if it is. encoding="utf-8" keeps
every character correct; UTF-8 is the most common way of writing the characters of every
language to disk. The loop then writes each sentence followed by "\n", the newline
character, so every sentence starts on its own line. Opening panadol.txt afterwards
is the easiest way to check what the program found on the page.
The whole program
The four steps together, 32 lines in all. The last three lines print a short report so you can see the result without opening the file:
import re
import requests
from bs4 import BeautifulSoup
url = "https://www.panadol.com/en-gb/adult-symptoms/headache/"
headers = {"User-Agent": "Mozilla/5.0 (compatible; SkillBot/1.0)"}
resp = requests.get(url, headers=headers, timeout=15)
resp.raise_for_status()
resp.encoding = resp.apparent_encoding
soup = BeautifulSoup(resp.text, "html.parser")
for tag in soup(["script", "style", "noscript", "nav", "footer"]):
tag.decompose()
lines = [" ".join(tag.get_text(" ").split()) for tag in soup.find_all(["h1", "h2", "h3", "p", "li"])]
lines = [line for line in lines if line != ""]
sentences = []
for line in lines:
for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
sentences.append(sentence)
lines = sentences
with open("panadol.txt", "w", encoding="utf-8") as outfile:
for line in lines:
outfile.write(line + "\n")
print(f"Saved {len(lines)} sentences to panadol.txt")
print("The first three sentences:")
for line in lines[:3]:
print(line)
The report lines use an f-string: a string with {...} blanks that
Python fills in, so {len(lines)} becomes the number of sentences.
lines[:3] takes the first three items of the list.
Running it on your own computer
Download the zip file and unzip
it to a folder. The zip holds the program and a pyproject.toml project file; there
is no input file, because the input is the webpage itself. Open the folder in Visual Studio
Code, then type these three lines in the terminal, one at a time:
uv python pin 3.12.10
uv sync
uv run webpage_sentences.py
The first line records which Python version this folder uses. The second line sets up the
project and installs the two packages the program needs, requests and
beautifulsoup4. The third line runs the program. The output looks like this:
Saved 307 sentences to panadol.txt
The first three sentences:
For Healthcare Professionals
United Kingdom
ASIA PACIFIC
The exact number may be different when you run it, because websites update their pages from time
to time. And notice what the first three sentences are: they come from the country selector at
the very top of the page, not from the headache article. That is a first hint that the
extraction is good but not perfect, which the next section discusses. Open
panadol.txt to see all the sentences the program saved. To read a different page,
change the url line and the output filename.
If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.
Limitations
- Some navigation text slips through. The country selector at the top of the
Panadol page is not inside a
navtag, so its words survive the cleanup, as the output above shows. Every website arranges its HTML differently, so a cleanup list that works for one page may leave stray bits on another. - Some pages refuse or mislead scrapers. A site can block requests that do not come from a real browser, and some pages build their content with JavaScript after the download, so the downloaded HTML never contains the article at all. Websites can also forbid scraping in their terms of use, so scrape gently: a page here and there, not thousands of pages in a loop.
- Sentence cutting has limits. The regular expression splits well on typed English but trips on abbreviations such as Dr. and on lowercase after punctuation; the details are in the sentence-splitting note.
In the age of AI, it has become a question of whether a website is read at all. Many users no longer visit websites themselves; they simply ask an AI model and read the answer it gives. If a company or an individual blocks every program from reading their pages, AI models cannot read those pages either: the content will never shape an AI's answer, and the site will never be named as a citation. That is why Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) — making your pages welcoming to AI searches, not only to traditional search engines — are becoming essential for any company or individual who wants a presence on the internet. Websites that shut out every program may feel safe, but they end up as lonely islands in the internet world: online, yet unseen by AI and missing from every AI answer.
None of these scraping problems is fatal: for most ordinary article pages the four steps produce a clean list of sentences ready for the next stage of work.
Further reading
- The requests documentation describes headers, timeouts and encodings in detail.
- The Beautiful Soup documentation lists all the ways to search a page, far beyond the two used here.
- How does a program split a text into sentences? explains the regular expression in step 3 one piece at a time.
- How to mitigate AI hallucination with Retrieval-Augmented Generation (RAG) by providing a webpage? shows what the saved sentences are good for: feeding them to an AI model so its answers can be traced back to real lines.