How does web scraping turn a webpage into a file of sentences?

Howard Pong Yuen Lam · AI Notes · November 3, 2026 · 6 min read

English 繁體中文 简体中文

A webpage is not plain text. Behind what you see, the page is a long piece of HTML code (HyperText Markup Language): the readable article is mixed together with the menu, the footer, the styling, and the scripts that make the page interactive. Before a program can do anything useful with a webpage — count its sentences, search it, or feed it to an AI model — it first has to pull the readable text out of that code.

Downloading a page and extracting its text with a program is called web scraping: the program scrapes the useful content off the page, leaving the code behind. The program in this note is a small web scraper that works in four steps: it downloads a webpage, keeps only the parts worth reading, cuts the text into sentences, and saves the sentences to a text file, one sentence per line. The example page is Panadol's page about headaches, and the output file is called panadol.txt.

Step 1: Downloading the page

url = "https://www.panadol.com/en-gb/adult-symptoms/headache/"
headers = {"User-Agent": "Mozilla/5.0 (compatible; SkillBot/1.0)"}
resp = requests.get(url, headers=headers, timeout=15)
resp.raise_for_status()
resp.encoding = resp.apparent_encoding

The requests library downloads pages from the internet. The first two lines set up the call: url is the web address, and headers carries a User-Agent, a short name that tells the website who is asking. Many websites refuse requests that do not look like they came from a browser, so the program introduces itself with a browser-like name.

requests.get(...) downloads the page; resp is short for response, the server's answer, and timeout=15 gives up after 15 seconds instead of waiting forever. raise_for_status() stops the program with a clear error if the server reported a failure, such as “page not found”, instead of quietly saving an error page as if it were the article.

The last line is about encoding: the rule that turns the page's raw bytes into readable characters. Web pages often declare their encoding wrongly, so the program switches to apparent_encoding, the encoding requests guessed from the actual bytes. This keeps special characters correct when the text is read.

Step 2: Keeping only the readable text

soup = BeautifulSoup(resp.text, "html.parser")
for tag in soup(["script", "style", "noscript", "nav", "footer"]):
    tag.decompose()

lines = [" ".join(tag.get_text(" ").split()) for tag in soup.find_all(["h1", "h2", "h3", "p", "li"])]
lines = [line for line in lines if line != ""]

BeautifulSoup, from the bs4 package (Beautiful Soup 4), parses the HTML code into a tree of tags that the program can search; the result is named soup. The next two lines throw away five kinds of tag we do not want: script (JavaScript code), style (page styling), noscript (text shown when JavaScript is off), nav (the menu) and footer (the bottom of the page). decompose() deletes each tag and everything inside it, so only the real content is left.

Then find_all collects the tags worth reading: headings (h1 is the largest, h3 smaller), paragraphs (p) and list items (li). For each tag, get_text(" ") takes out just the words, and " ".join(... .split()) squeezes any run of spaces and line breaks into single spaces, so each tag becomes one tidy line of text. The second line drops empty lines, keeping only lines that really contain words.

Step 3: Cutting the lines into sentences

sentences = []
for line in lines:
    for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
        sentences.append(sentence)
lines = sentences

One line can hold several sentences, so the program cuts each line wherever a space comes after a full stop, exclamation mark or question mark and is followed by a capital letter. The pattern is a regular expression, a search pattern handled by Python's re module, and the five lines around it are explained piece by piece in How does a program split a text into sentences? After these lines, the name lines refers to sentences, not webpage lines.

Step 4: Saving the sentences to a file

with open("panadol.txt", "w", encoding="utf-8") as outfile:
    for line in lines:
        outfile.write(line + "\n")

open("panadol.txt", "w", ...) opens the file in the same folder for writing, creating it if it is not there and emptying it if it is. encoding="utf-8" keeps every character correct; UTF-8 is the most common way of writing the characters of every language to disk. The loop then writes each sentence followed by "\n", the newline character, so every sentence starts on its own line. Opening panadol.txt afterwards is the easiest way to check what the program found on the page.

The whole program

The four steps together, 32 lines in all. The last three lines print a short report so you can see the result without opening the file:

import re

import requests
from bs4 import BeautifulSoup

url = "https://www.panadol.com/en-gb/adult-symptoms/headache/"
headers = {"User-Agent": "Mozilla/5.0 (compatible; SkillBot/1.0)"}
resp = requests.get(url, headers=headers, timeout=15)
resp.raise_for_status()
resp.encoding = resp.apparent_encoding

soup = BeautifulSoup(resp.text, "html.parser")
for tag in soup(["script", "style", "noscript", "nav", "footer"]):
    tag.decompose()

lines = [" ".join(tag.get_text(" ").split()) for tag in soup.find_all(["h1", "h2", "h3", "p", "li"])]
lines = [line for line in lines if line != ""]

sentences = []
for line in lines:
    for sentence in re.split(r"(?<=[.!?])\s+(?=[A-Z])", line):
        sentences.append(sentence)
lines = sentences

with open("panadol.txt", "w", encoding="utf-8") as outfile:
    for line in lines:
        outfile.write(line + "\n")

print(f"Saved {len(lines)} sentences to panadol.txt")
print("The first three sentences:")
for line in lines[:3]:
    print(line)

The report lines use an f-string: a string with {...} blanks that Python fills in, so {len(lines)} becomes the number of sentences. lines[:3] takes the first three items of the list.

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the program and a pyproject.toml project file; there is no input file, because the input is the webpage itself. Open the folder in Visual Studio Code, then type these three lines in the terminal, one at a time:

uv python pin 3.12.10
uv sync
uv run webpage_sentences.py

The first line records which Python version this folder uses. The second line sets up the project and installs the two packages the program needs, requests and beautifulsoup4. The third line runs the program. The output looks like this:

Saved 307 sentences to panadol.txt
The first three sentences:
For Healthcare Professionals
United Kingdom
ASIA PACIFIC

The exact number may be different when you run it, because websites update their pages from time to time. And notice what the first three sentences are: they come from the country selector at the very top of the page, not from the headache article. That is a first hint that the extraction is good but not perfect, which the next section discusses. Open panadol.txt to see all the sentences the program saved. To read a different page, change the url line and the output filename.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

Limitations

In the age of AI, it has become a question of whether a website is read at all. Many users no longer visit websites themselves; they simply ask an AI model and read the answer it gives. If a company or an individual blocks every program from reading their pages, AI models cannot read those pages either: the content will never shape an AI's answer, and the site will never be named as a citation. That is why Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) — making your pages welcoming to AI searches, not only to traditional search engines — are becoming essential for any company or individual who wants a presence on the internet. Websites that shut out every program may feel safe, but they end up as lonely islands in the internet world: online, yet unseen by AI and missing from every AI answer.

None of these scraping problems is fatal: for most ordinary article pages the four steps produce a clean list of sentences ready for the next stage of work.

Further reading