A Wayback Machine scraper is a script or tool that pulls content out of archived web pages at scale, for example every product name from an old shop, every article from a closed blog, or every version of a price list over several years. The reliable way to build one has two parts: first ask the archive’s CDX index which captures exist, then fetch each capture’s original content and extract the data you need. Done carefully, with pauses between requests and a clear purpose, it turns thousands of archived pages into a usable spreadsheet or content export.
This guide explains how a Wayback Machine scraper (or any web archive scraper) works, gives a simple Python example, compares ready-made tools, and covers the limits and etiquette that keep a project on the right side of the archive’s rules. If you want the pages themselves as a working website rather than extracted data, our Wayback website download service does that without any code.
Scraping versus downloading
| Scraping | Downloading | |
|---|---|---|
| Goal | Extract specific data from many pages | Save pages and assets as files |
| Output | Spreadsheet, JSON or database rows | HTML, images, CSS and scripts |
| Typical use | Research, content recovery, price history | Restoring or hosting a site |
| Tools | Python scripts, Scrapy, waybackpy | Whole-site downloaders, download services |
If your goal is a copy of the site, read our guide on how to use Wayback Machine Downloader instead, or have a Wayback Machine website downloader service hand back cleaned files.
How a Wayback Machine scraper works
1. Decide exactly what you are collecting
Write down the fields, the pages and the period before touching any code. “Every product name and price on example.com/shop/ between 2016 and 2019” is a scraping project. “Everything on the site” is a download project, and a scraper is the wrong tool for it. A narrow brief keeps the job small, fast and polite.
2. Get the list of captures
Ask the archive’s index which pages were captured in your period, keeping only successful HTML pages and one capture per page. Our Wayback Machine API guide covers the query options, and the Internet Archive documents every parameter in the CDX server README. The Python example below shows a typical request a Wayback Machine scraper sends.
3. Fetch the original content of each capture
Archived pages are normally served with the archive’s toolbar and rewritten links. Adding id_ after the timestamp in the archived address returns the page as it was captured, which is much easier to parse.
4. Extract the data
Parse the HTML with a library such as BeautifulSoup or lxml and pull out the fields you need: titles, headings, body text, prices, dates, image addresses. Because old sites often changed their templates, test your selectors on captures from several years before running the full job.
5. Store the results with their source
Save each row with the capture timestamp and archived address. That record shows exactly where each value came from, which matters for research and for any content you later republish.
What people use a Wayback Machine scraper for
- Recovering blog content. Pull the title, date, author and body text of every post from a closed blog into a CSV, ready to import into a new CMS.
- Rebuilding a product catalogue. Extract product names, descriptions and image addresses from an old shop so the range can be re-entered on a new store.
- Price and content history. Track how a page’s prices, wording or offers changed across captures for research or competitive analysis.
- SEO history. Collect old page titles, meta descriptions and headings to see how a site was optimised before it was lost.
- Academic and journalism research. Build a dated dataset of statements published on a site, with the archived address as evidence.
Cleaning scraped archive data
Raw results from archived pages are rarely ready to use. Expect to remove duplicates where the same page was captured under slightly different addresses, fix character encoding problems on very old pages, strip navigation and footer text that slipped into the body, and normalise dates written in different formats. Keep the uncleaned export alongside the cleaned version so you can check any value against its source later.
A simple Python example
import time
import requests
from bs4 import BeautifulSoup
HEADERS = {"User-Agent": "archive-research/1.0 ([email protected])"}
CDX = "https://web.archive.org/cdx/search/cdx"
params = {
"url": "example.com/blog/*",
"output": "json",
"fl": "timestamp,original",
"filter": ["statuscode:200", "mimetype:text/html"],
"collapse": "urlkey",
}
rows = requests.get(CDX, params=params, headers=HEADERS, timeout=60).json()[1:]
results = []
for timestamp, original in rows:
page = f"https://web.archive.org/web/{timestamp}id_/{original}"
html = requests.get(page, headers=HEADERS, timeout=60).text
soup = BeautifulSoup(html, "html.parser")
title = soup.find("h1") or soup.find("title")
results.append({"url": original, "captured": timestamp,
"title": title.get_text(strip=True) if title else ""})
time.sleep(2) # be gentle with the archiveThis collects the title of every archived blog post with its source. Swap the extraction lines for the fields your project needs, and write the results to CSV or JSON at the end.
Ready-made web archive scraper tools
- Scrapy-based Wayback tools. There are open-source Scrapy middleware and command-line projects that route a Scrapy spider through Wayback captures, which suits developers who already use Scrapy.
- Waybackpy. A Python library for the archive’s APIs; useful for building the capture list without writing the requests yourself. See our waybackpy guide.
- Waybackpack. Downloads every capture of a single URL, handy when you want the full history of one page before extracting from it. See our waybackpack guide.
- Whole-site downloaders. If you need files rather than data, compare options in our overview of Wayback Machine downloader alternatives.
Scraping responsibly
- Go slowly. Pause between requests and run one job at a time. If you receive HTTP 429 responses, you are being rate limited; increase the delay.
- Identify yourself. Use a user agent that names your project and gives a contact address.
- Cache everything. Save each fetched page to disk so reruns do not hit the archive again.
- Collect only what you need. Narrow the CDX query by path and date instead of fetching the whole domain.
- Respect content rights and privacy. Archived pages remain their creators’ work. Scraping your own old site is straightforward; republishing third-party content or collecting personal data needs a lawful basis.
Common problems and fixes
Many pages return nothing useful
Some captures are redirects, error pages or cookie walls saved by the crawler. Filter by status code and check a sample by eye before trusting the results.
Content loaded by JavaScript is missing
If the original site loaded text with JavaScript after the page opened, the HTML capture may not contain it. Look for the data files the page requested, which are sometimes archived separately, or accept the gap.
Selectors break across years
Sites change templates. Group captures by period and write a selector for each template, or fall back to broader text extraction.
Images in your data do not load
Image addresses in archived pages point to captures that may not exist. Check them individually; our guide to recovering missing images from the Wayback Machine explains where else to look.
Frequently asked questions
Is it allowed to scrape the Wayback Machine?
The archive provides public APIs for programmatic access, and reasonable, rate-limited use for research or recovering your own content is common. Heavy or abusive traffic can be blocked, and the content itself is still subject to its owners’ rights.
Do I need to know Python?
For a custom scraper, basic Python is enough. If you only want the old pages back as files, a downloader or a download service is simpler.
Can a scraper rebuild my old website?
It can recover the content, but not a working site. Rebuilding needs the pages, images and styles downloaded and cleaned, which is a different job.
Not sure which route fits your project? Request a free archive review and we will look at what the archive holds for your domain.