Wget Wayback Machine Downloads: The Method That Actually Works

You can use wget to download pages from the Wayback Machine, but pointing wget -r at an archived page is usually a mistake: the recursive crawl follows the archive’s rewritten links across many capture dates, downloads the toolbar and duplicate copies, and quickly runs into rate limits. The method that works is a wget Wayback Machine download from a list: first get the list of captures from the archive’s CDX index, turn each one into an id_ address that returns the original file, and let wget fetch that list slowly with --input-file. You end up with the original HTML, images and files, organised by path, ready for cleanup.

This guide walks through that method on Linux, macOS and Windows, explains the wget options that matter, and shows what still needs fixing afterwards. If you would rather receive a cleaned, working copy, our Wayback website download service does the whole job.

Why recursive wget struggles with the archive

• Rewritten links: archived pages link to other archived addresses with their own timestamps, so a recursive crawl jumps between dates and saves near-identical copies.
• Archive extras: the toolbar, scripts and banner code are downloaded with every page.
• Volume: one site can expand into many thousands of requests, which triggers rate limiting and errors.
• Messy folders: files land under long archive paths instead of the site’s own structure.

Working from a capture list avoids all four problems because you decide exactly which files to fetch.

Step 1: Install wget

• Linux: usually installed; otherwise use your package manager.
• macOS: install it with a package manager such as Homebrew.
• Windows: install a Windows build of GNU wget through a package manager, or use wget inside WSL. Note that in Windows PowerShell 5.1, typing wget runs PowerShell’s own web command, not GNU wget, so call wget.exe explicitly or use a newer PowerShell where the alias is gone.

Check the install with wget --version. Every option used below is described in the GNU wget manual.

Step 2: Get the list of captures

Ask the CDX index for one capture per file in the period you want. This example lists every successful capture on a domain during 2019, one per address, as plain text with the timestamp and original URL:

https://web.archive.org/cdx/search/cdx?url=example.com/*&from=2019&to=2019&fl=timestamp,original&filter=statuscode:200&collapse=urlkey

Open it in a browser or fetch it with wget itself and save the result as captures.txt:

wget -O captures.txt "https://web.archive.org/cdx/search/cdx?url=example.com/*&from=2019&to=2019&fl=timestamp,original&filter=statuscode:200&collapse=urlkey"

Leaving out a content-type filter means images, stylesheets and PDFs are included, which you want for a usable copy. For more query options, see our Wayback Machine API guide. To choose a good year first, read how to choose a Wayback snapshot.

Step 3: Turn captures into original-file addresses

Each line of captures.txt holds a timestamp and an address. Adding id_ after the timestamp asks the archive for the file exactly as captured, without the toolbar or rewritten links.

On Linux or macOS:

sed -E 's#^([0-9]+) (.*)#https://web.archive.org/web/\1id_/\2#' captures.txt > urls.txt

In PowerShell:

Get-Content captures.txt | ForEach-Object { $t, $u = $_ -split ' ', 2; "https://web.archive.org/web/${t}id_/$u" } | Set-Content urls.txt

Step 4: Run the wget Wayback Machine download

wget --input-file=urls.txt \
     --force-directories --no-host-directories --cut-dirs=3 \
     --wait=2 --random-wait --tries=3 --no-clobber \
     --user-agent="site-recovery/1.0 ([email protected])"

Option	What it does
--input-file	Downloads exactly the addresses in your list, nothing more
--force-directories	Recreates the folder structure from each address
--no-host-directories	Drops the web.archive.org folder
--cut-dirs	Removes the leading archive path parts so folders start at your domain; check the first results and adjust the number if needed
--wait and --random-wait	Pause between requests to stay polite and avoid rate limits
--tries	Retries a failed file a few times
--no-clobber	Skips files already downloaded, so you can resume
--user-agent	Identifies your job with a contact address

On Windows, write the command on one line or replace the backslashes with PowerShell’s backtick line continuation.

Downloading only part of a site

You rarely need everything. To fetch just the blog, change the address in the capture query to example.com/blog/*; to fetch only images, add a content-type filter such as filter=mimetype:image/jpeg. A smaller list means a faster, gentler job and less cleanup. You can always run a second list later for anything you missed, and --no-clobber stops wget from fetching files you already have.

Step 5: Clean up the downloaded files

The download gives you original files, but the site will not work offline or on hosting straight away:

• Links still point to the original domain. Pages refer to https://example.com/... addresses. wget’s --convert-links option cannot fix this here, because the downloaded files came from archive addresses, not the original ones. Rewrite them to relative paths or to the new domain; our guide on fixing links in a downloaded Wayback website shows how.
• Some files were never captured. Missing images or stylesheets show up as broken parts of the page. See recovering missing images from the Wayback Machine and recreating missing CSS.
• Folder pages need index files. Addresses ending in a slash may have been saved as index.html, which works on most servers but should be checked.
• Query-string pages. Addresses with parameters are saved with unusual file names and may need renaming or redirects.

Common wget errors with the Wayback Machine

429 Too Many Requests

You are going too fast. Increase --wait, keep --random-wait, and run one job at a time. The Wayback Machine 429 Too Many Requests guide explains why the archive blocks fast clients and how long to wait before resuming.

Downloads of HTML that look like the archive page

The address is missing id_. Check that every line in urls.txt has id_ right after the timestamp.

Folders start with long archive paths

Adjust --cut-dirs. Look at where the first file was saved and increase or decrease the number until the folders begin with your domain or its pages.

Some files return errors

Some captures fail or redirect even when listed. Collect the failures from the log, look for other capture dates of the same file, and fetch those instead. More fixes are in our guide to a Wayback Machine downloader that is not working.

wget compared with dedicated downloaders

wget gives you full control and is already installed on many systems, which makes it a good choice for technical users who want a specific set of files. Dedicated Wayback downloaders automate the listing and path handling, which saves time on large sites. Neither produces a finished website on its own; the cleanup in step 5 applies to every method. Our overview of Wayback Machine downloader alternatives compares the options, and our own Wayback Machine downloader service handles the listing, retries and cleanup for you.

Frequently asked questions

Can wget download a whole website from the Wayback Machine?

Yes, if you give it the list of captured files rather than letting it crawl. Recursive mode against the archive produces duplicates and archive code.

What does id_ do in a Wayback address?

It asks the archive for the captured file as it originally was, without the toolbar or rewritten links, which is what you want when saving files.

Does wget work on Windows?

Yes, with a Windows build of GNU wget or inside WSL. In older PowerShell, call wget.exe so you do not get PowerShell’s built-in command.

Prefer a cleaned copy without the command line? Request a free archive review and we will tell you what can be recovered.