waybackurls: How to List Every Archived URL for a Domain

waybackurls is a small, free command-line tool, written in Go by Tom Hudson (tomnomnom), that prints every URL the Wayback Machine knows about for a domain. You give it a domain, it asks the archive’s CDX index, and it returns a plain list of archived addresses that you can filter, count or feed into a downloader. It is popular with security researchers, but it is just as useful when you own an old website and need a full list of its former pages before you download or rebuild it. This guide shows how to install waybackurls, the three flags it has, how to clean its output, and where its limits are.

What waybackurls does (and what it does not)

waybackurls answers one question: which URLs under this domain were ever captured? It does not download pages, images or files. It does not tell you whether a capture is a working page or an error. It simply lists addresses.

Reading the tool’s source code on GitHub shows exactly where the list comes from:

• The Wayback Machine CDX index. It requests web.archive.org/cdx/search/cdx with a wildcard for the domain (and its subdomains by default), JSON output, and collapse=urlkey so each URL appears once.
• A Common Crawl index. The code also queries a Common Crawl index. The index name is fixed in the source, so treat those results as an older extra source, not a current crawl.
• VirusTotal, only if you set a key. If an environment variable called VT_API_KEY is present, it also asks VirusTotal for URLs it has seen on the domain. Without the key this source is skipped.

That is why a waybackurls list can contain a few addresses that you will not find on the Wayback Machine calendar: they came from one of the other sources.

How to install waybackurls

waybackurls is distributed as Go source code. You need the Go toolchain installed (from the official Go website), then run:

go install github.com/tomnomnom/waybackurls@latest

Go places the compiled binary in its bin folder (usually ~/go/bin on macOS and Linux, or %USERPROFILE%\go\bin on Windows). Add that folder to your PATH, open a new terminal, and check it runs:

waybackurls -h

On Windows the same command works in PowerShell or Command Prompt once Go is installed. The pipe examples below use Unix tools such as grep and sort; on Windows you can run them in Git Bash or WSL, or use PowerShell equivalents such as Select-String and Sort-Object -Unique.

Basic waybackurls usage

There are two ways to pass a domain. As an argument:

waybackurls example.com > urls.txt

Or through standard input, which lets you process several domains in one go:

cat domains.txt | waybackurls > urls.txt

The output is one URL per line. For a small site that may be a few hundred lines; for an old blog or shop it can be tens of thousands, because every query-string variation, feed, image and script that was captured counts as a separate URL.

The three waybackurls flags

Flag	What it does	When to use it
-no-subs	Excludes subdomains; only the exact domain you gave is searched	When shop., mail. or cdn. subdomains add noise you do not need
-dates	Adds the capture date in the first column	When you want to see which years a page existed, for example to pick a snapshot
-get-versions	For an input URL (not a domain), lists the archived capture URLs of that page	When you need every saved version of one page, such as the homepage

Example of the version listing for a single page:

echo "https://example.com/about/" | waybackurls -get-versions

Each line it returns is a Wayback address with a timestamp. The source builds these with the if_ modifier, which replays the page without the full archive toolbar frame.

Cleaning a waybackurls list so it is actually useful

A raw list mixes real pages with assets, tracking parameters and junk. If your goal is to rebuild or download a site, reduce it to the pages that matter.

1. Remove duplicates. sort -u urls.txt > unique.txt
2. Drop static files when you want pages only: grep -viE '\.(jpg|jpeg|png|gif|svg|webp|css|js|woff2?|ico)(\?|$)' unique.txt > pages.txt
3. Strip query strings that only differ by tracking parameters: sed 's/[?#].*$//' pages.txt | sort -u > clean.txt. Check first that the site did not use query strings for real pages (for example ?p=123 on older WordPress sites).
4. Group by folder to see the site’s structure: cut -d/ -f4 clean.txt | sort | uniq -c | sort -rn | head
5. Keep a separate asset list (images, PDFs) if you plan to download files later.

After cleaning, a list of 20,000 raw URLs often comes down to a few hundred real pages, which is a much better basis for planning a download.

Check which URLs are real pages before you rely on them

waybackurls does not show HTTP status codes. Many listed URLs were captured as redirects, 404 pages or server errors. To see the status of each capture, query the CDX index directly and include the status field. Our guide to the Wayback Machine CDX API shows how to request original, timestamp and statuscode and filter to statuscode:200.

A practical approach is to use waybackurls for a quick overview, then use a single CDX query with a status filter for the list you will actually download. That way you avoid spending requests on captures that were never real pages.

Using the list to rebuild or download a website

A clean URL list is the starting point for most do-it-yourself projects:

• Rebuilding a sitemap and redirects. The list tells you which old addresses had links and traffic, so you can recreate them or redirect them. See how to find historical URLs and rebuild a sitemap for the full method.
• Downloading the pages. waybackurls lists; it does not fetch. To save the captured files, use a downloader such as the open-source wayback-machine-downloader (see how to use Wayback Machine Downloader) or wget with the id_ modifier (see our wget Wayback Machine guide).
• Estimating the size of the job. The number of clean page URLs, plus the number of images and documents, is a fair indicator of how long a careful download will take.

Limits and good manners

• Rate limits still apply. waybackurls makes few requests per domain, but piping hundreds of domains through it, or running it in parallel, can trigger the archive’s protections. If you see errors, slow down; our guide to the Wayback Machine 429 error explains why.
• Listed does not mean recoverable. A URL in the list may point to a capture with missing images or styles, and the archive never holds databases, logins or server-side code.
• Very large domains. For huge sites a single CDX request can be slow or time out. Splitting the job by folder with the CDX API is more reliable.
• Maintenance. The repository has not changed for some time. It still works because the CDX interface it calls is stable, but check the issues page if output looks wrong.
• Use it responsibly. The tool is common in security research. Run it on domains you own or are authorised to assess, and respect the Internet Archive as a free public service.

waybackurls vs other ways to list archived URLs

Method	Good for	Trade-off
waybackurls	Fast list for one or many domains, easy to pipe	No status codes; extra sources mixed in
CDX API directly	Precise filters (status, MIME type, date range)	You write the query yourself
Wayback Machine web search with /*	Browsing a small site without installing anything	Manual, slow for large sites
Python libraries such as waybackpy	Scripts that need captures plus metadata	Requires Python setup

When a URL list is not enough

If you have the list but not the time to download, clean and relink thousands of files, you can check what the archive holds first with our free archive review, or have the captured site delivered as a clean, browsable HTML ZIP through the Wayback website download service. Either way, the result depends on what was captured; no tool can add pages the crawler never saved.

Frequently asked questions

Is waybackurls free?

Yes. Its source code is public on GitHub and you compile it with Go. The Wayback Machine data it reads is also free to access, within the archive’s rate limits.

Does waybackurls download the archived pages?

No. It only prints URLs. Use a downloader or wget with the list to save the files themselves.

Why does waybackurls return URLs I cannot open in the Wayback Machine?

Some URLs come from Common Crawl or, if configured, VirusTotal rather than the Wayback Machine. Others were captured as redirects or errors. Check the status with a CDX query.

How do I exclude subdomains in waybackurls?

Add the -no-subs flag, for example waybackurls -no-subs example.com.

Does waybackurls work on Windows?

Yes. Install Go for Windows, run the same go install command, and add the Go bin folder to your PATH.