How to Monitor Websites Automatically with Python
Websites change constantly. Prices are updated, product pages are modified, news headlines appear, job listings are added, and important announcements can be published without warning. Checking these changes manually can be time-consuming.
Python makes it possible to build a simple website monitoring system that regularly checks a webpage and alerts you when something changes. With a few libraries, even beginners can create a useful monitoring script.
What Is Website Monitoring?
Website monitoring means automatically checking a website or webpage at regular intervals and detecting changes.
For example, you could monitor:
- A product price
- A job listing
- A news page
- A blog post
- A website's availability
- A particular piece of text
- A public announcement
- Changes to HTML content
Instead of opening the website repeatedly, Python can perform these checks for you.
How Python Can Monitor a Website
A basic monitoring program follows this process:
Website → Download page → Extract information → Compare with previous result → Detect change → Send notification
Python provides several libraries that make this process relatively straightforward.
For a simple monitor, you can use:
requests— downloads webpage contentBeautifulSoup— extracts information from HTMLhashlib— creates a fingerprint of contenttime— waits between checkssmtplibor a notification service — sends alerts
Installing the Required Libraries
First, install the two commonly used packages:
pip install requests beautifulsoup4
Then create a Python file such as:
website_monitor.py
A Simple Website Change Monitor
Here is a basic example that checks whether the content of a webpage has changed:
import requests
import hashlib
import time
URL = "https://example.com"
def get_page_hash():
response = requests.get(
URL,
timeout=10,
headers={"User-Agent": "Mozilla/5.0"}
)
response.raise_for_status()
content = response.text
return hashlib.sha256(content.encode("utf-8")).hexdigest()
old_hash = get_page_hash()
print("Monitoring started...")
while True:
time.sleep(300) # Check every 5 minutes
try:
new_hash = get_page_hash()
if new_hash != old_hash:
print("Website has changed!")
old_hash = new_hash
else:
print("No changes detected.")
except requests.RequestException as error:
print("Unable to check website:", error)
The program downloads the webpage and creates a SHA-256 hash from its HTML. If the hash changes during the next check, the program reports that the page has changed.
Why Use a Hash?
Comparing an entire webpage every time can be inconvenient. A hash provides a compact representation of the content.
For example:
Webpage content
↓
SHA-256
↓
a8f4...9c21
If even a small part of the input changes, the resulting hash will normally be different.
This makes hashes useful for detecting whether a downloaded document or webpage has changed.
Monitoring Only a Specific Part of a Website
Sometimes you don't care about the entire webpage.
Imagine you want to monitor only a product price:
<div class="price">₹49,999</div>
Using BeautifulSoup, you can extract that particular element.
import requests
from bs4 import BeautifulSoup
URL = "https://example.com/product"
response = requests.get(
URL,
timeout=10,
headers={"User-Agent": "Mozilla/5.0"}
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
price = soup.select_one(".price")
if price:
print("Current price:", price.get_text(strip=True))
Now the monitoring system can compare the price rather than the complete webpage.
Building a Price Change Detector
The next step is to remember the previous value.
import requests
from bs4 import BeautifulSoup
import time
URL = "https://example.com/product"
def get_price():
response = requests.get(
URL,
timeout=10,
headers={"User-Agent": "Mozilla/5.0"}
)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
element = soup.select_one(".price")
return element.get_text(strip=True) if element else None
old_price = get_price()
print("Initial price:", old_price)
while True:
time.sleep(300)
try:
new_price = get_price()
if new_price != old_price:
print("Price changed!")
print("Old:", old_price)
print("New:", new_price)
old_price = new_price
else:
print("Price has not changed.")
except requests.RequestException as error:
print("Error:", error)
This example can form the foundation of a much larger monitoring application.
Sending an Alert
Printing a message in the terminal is useful for testing, but a monitoring system becomes much more useful when it sends a notification.
For example, you could connect your Python program to:
- Telegram
- Discord
- Slack
- A custom notification API
The basic workflow becomes:
Website changes
↓
Python detects change
↓
Notification function
↓
Your phone/email
A notification function might look like this:
def send_alert(message):
print("ALERT:", message)
You can later replace this function with an email or messaging API.
Monitoring Multiple Websites
Python can also monitor several pages.
websites = {
"News": "https://example.com/news",
"Blog": "https://example.com/blog",
"Jobs": "https://example.com/jobs"
}
for name, url in websites.items():
print("Checking:", name)
response = requests.get(
url,
timeout=10,
headers={"User-Agent": "Mozilla/5.0"}
)
print(response.status_code)
For a larger application, you could store the websites in a JSON file or database.
For example:
[
{
"name": "News",
"url": "https://example.com/news"
},
{
"name": "Jobs",
"url": "https://example.com/jobs"
}
]
This makes adding new websites easier without changing the Python program.
Using SQLite for Persistent Monitoring
A more advanced monitor should remember previous results even after the program closes.
Python includes SQLite support through the built-in sqlite3 module.
You could store:
- Website URL
- Last checked time
- Previous content hash
- Last detected change
- Monitoring status
A simplified database structure might look like:
websites
-------------------------
id
name
url
content_hash
last_checked
This turns a small script into the foundation of a real monitoring service.
Handling Websites That Use JavaScript
One important limitation is that requests downloads the server's response but does not behave like a normal browser.
Some websites generate their content using JavaScript.
In such cases, BeautifulSoup may not find the information you're looking for because the desired content isn't present in the initial HTML.
For websites that require browser rendering, tools such as Playwright or Selenium can be used.
A browser-based workflow looks like:
Python
↓
Automated browser
↓
Load JavaScript
↓
Rendered webpage
↓
Extract information
↓
Compare with previous result
However, browser automation requires more resources than a simple requests-based monitor.
Important Monitoring Practices
Website monitoring should be performed responsibly.
Before monitoring a website, check its terms of service and robots.txt where applicable. Avoid sending excessive requests because frequent automated requests can put unnecessary load on a server.
A good monitoring application should:
- Use reasonable intervals
- Set request timeouts
- Handle errors
- Identify itself appropriately where appropriate
- Avoid bypassing authentication or access controls
- Respect website terms and applicable laws
- Cache information when possible
For many tasks, checking every few minutes or even every hour is sufficient.
Turning the Script Into a Background Service
Once your monitor works, you don't necessarily need to keep a terminal window open manually.
On Linux, you can use:
cron
On Windows, you can use:
Task Scheduler
You can also deploy the monitoring program to a server or cloud environment.
Another approach is to use Python's scheduling libraries to execute monitoring jobs at predefined intervals.
Taking the Project Further
A basic website monitor can evolve into a complete application.
You could build a dashboard using Flask, FastAPI, or Django.
The dashboard could display:
Website Monitor
--------------------------------
Website Status Last Check
News Changed 10:30 PM
Jobs Stable 10:29 PM
Product Changed 10:28 PM
Blog Stable 10:27 PM
You could also add:
- User accounts
- Website categories
- Custom CSS selectors
- Change history
- Email notifications
- Telegram notifications
- Scheduling
- Database storage
- Dashboard charts
- Automatic screenshots
- Failure alerts
- Multiple monitoring frequencies
Conclusion
Python provides a practical way to automate website monitoring without building a complicated system from scratch. A simple combination of requests, BeautifulSoup and hashing can detect basic webpage changes, while tools such as Playwright can handle more dynamic websites.
The most important idea is to monitor only the information you actually need. Instead of repeatedly comparing an entire webpage, extracting a specific price, headline, status, or section usually produces cleaner and more useful results.
With persistent storage and notifications, a small Python script can eventually become a powerful website monitoring platform that works continuously in the background.
