Wayback Machine API: How to Use the CDX and Availability APIs

The Wayback Machine API is really a small family of free, public endpoints run by the Internet Archive. The two you will use most are the Availability API, which tells you whether a page has a capture near a given date, and the CDX API, which lists every capture of a URL, a folder or a whole domain. Neither needs an account or a key for public data. With them you can find out what the archive holds before you download anything, build a list of old URLs, and fetch the original files without the archive toolbar. This guide shows the requests, the parameters that matter, and the limits to plan around.

What the Wayback Machine API can and cannot do

The API reads the same index that powers the calendar view on web.archive.org. It returns data about captures: the URL, the timestamp, the MIME type, the HTTP status the crawler received, a content digest and the stored size. It then lets you request any listed capture.

• It can tell you whether a page was ever captured, list every captured URL under a domain, filter by date, file type or status code, and serve the stored copy of each file.
• It cannot return anything the crawler never saved. There is no database, no server code, no admin area and no page that sat behind a login. If a URL is not in the index, no request will produce it.
• It is not built for bulk mirroring at speed. The Internet Archive rate-limits heavy clients, and fast scripts start receiving HTTP 429 (Too Many Requests) responses.

The official reference for the CDX server lives in the Internet Archive’s wayback-cdx-server documentation, and the Availability API is described on the Wayback Machine APIs help page.

The Availability API: is there a capture near this date?

Use this when you have one URL and want the closest capture to a date. It is the quickest check of all.

https://archive.org/wayback/available?url=example.com&timestamp=20150101

The response is JSON. If a capture exists, archived_snapshots.closest contains available, the full url of the capture, its timestamp and the status code. If nothing exists the object is empty. Leave out timestamp to get the most recent capture. The timestamp can be as short as a year (2015) or as long as fourteen digits (20150101120000).

The Availability API returns only one capture, so it is a yes/no check. To see everything that was saved, move to the CDX API. If you only need a quick list of every archived URL for a domain, the waybackurls command-line tool wraps this query for you.

The CDX API: list every capture

The CDX endpoint is https://web.archive.org/cdx/search/cdx. The only required parameter is url. A simple request looks like this:

https://web.archive.org/cdx/search/cdx?url=example.com/about.html&output=json

With output=json you get an array whose first row names the columns. By default each row holds seven fields in this order: urlkey, timestamp, original, mimetype, statuscode, digest and length. Without output=json the same data comes back as space-separated text, which is handy for command-line tools.

Parameters that matter for a site download

• matchType: exact (the default) returns one URL; prefix returns everything under a path; host returns everything on one hostname; domain includes subdomains. A trailing /* on the URL also means prefix, so url=example.com/blog/* lists the whole blog.
• from and to: an inclusive date range, from 1 to 14 digits. from=2014&to=2016 keeps only captures from those years.
• fl: the fields to return, for example fl=timestamp,original,statuscode. Fewer fields means smaller responses.
• filter: a regular-expression filter on a field, written field:regex. Prefix it with ! to exclude. filter=statuscode:200 keeps successful captures; filter=!mimetype:image/.* removes images. You can repeat filter several times.
• collapse: drops adjacent duplicates. collapse=urlkey gives one row per unique URL; collapse=digest skips captures whose content did not change; collapse=timestamp:8 keeps at most one capture per day.
• limit: caps the number of rows. A negative value such as limit=-1 returns the last rows, which can be slow on large URLs.
• showResumeKey and resumeKey: for large result sets, ask for a resume key, then send it back with the same parameters to continue where the previous response stopped.

Wayback Machine API recipes you can copy

Replace example.com with the domain you are researching.

Goal	Request
Every unique URL on the site	/cdx/search/cdx?url=example.com&matchType=host&fl=original&collapse=urlkey
Only HTML pages that returned 200	...&filter=statuscode:200&filter=mimetype:text/html&collapse=urlkey
All PDFs ever captured	...&filter=mimetype:application/pdf&collapse=urlkey
How a single page changed over time	/cdx/search/cdx?url=example.com/pricing.html&collapse=digest&fl=timestamp,statuscode
Captures from one period only	...&from=2018&to=2019
Count result pages before a huge query	...&showNumPages=true

The URL list from the first recipe is also the raw material for redirect maps and sitemaps; our guide to finding historical URLs and rebuilding a sitemap explains how to clean and classify it.

Download the original file without the archive toolbar

Once a CDX row gives you a timestamp and an original URL, you can request the stored copy:

https://web.archive.org/web/20150101000000/http://example.com/about.html

That normal replay URL adds the Wayback toolbar and rewrites links so they point back into the archive. For a clean copy, add id_ straight after the timestamp:

https://web.archive.org/web/20150101000000id_/http://example.com/about.html

The id_ form returns the file as it was captured, without the injected toolbar script and without rewritten links. It is the form most downloaders use under the hood. The timestamp does not have to be exact: the archive redirects to the nearest capture, so check the final URL if the date matters.

A minimal script: list pages, then fetch them politely

Here is the logic in plain steps. It works the same in Python, Node.js, PowerShell or a shell script.

1. Request the CDX list for the host with fl=timestamp,original, filter=statuscode:200 and collapse=urlkey.
2. Choose one timestamp per URL. The latest capture before the site changed owners is often the right one; the snapshot selection guide covers how to pick.
3. For each row, request /web/{timestamp}id_/{original} and save it under a path built from the original URL.
4. Wait between requests. One request every second or two is a reasonable starting point; back off further when you see HTTP 429 or connection resets.
5. Log every failure and retry those later instead of restarting the whole job.

With wget, you can feed a file of id_ URLs into wget --input-file=urls.txt --wait=2 --random-wait --force-directories. Do not point wget --mirror at a normal replay URL: it follows the rewritten links across dates and pulls in toolbar files you do not want. Our step-by-step guide to using wget with the Wayback Machine covers the full command and its flags. If you need text or fields from archived pages rather than whole files, see how a Wayback Machine scraper works.

Rate limits, errors and etiquette

The Internet Archive is a non-profit library, and its servers are shared by everyone. Heavy parallel requests slow it down for others and get your client throttled.

• HTTP 429 means you are sending too much. Pause, lower concurrency to one, and resume.
• HTTP 503 or timeouts can happen during busy periods. Retry with a growing delay.
• Empty CDX responses usually mean the URL form is wrong. Try with and without www, with matchType=host, or with a trailing /*.
• Huge result sets are capped per query. Narrow by path, date or MIME type, or use resume keys instead of one giant request.
• Set a descriptive User-Agent with contact details in scripts, so the archive can reach you rather than block you.

If your script stalls, the downloader troubleshooting guide lists the common causes and fixes.

When the API is enough, and when it is not

The Wayback Machine API is ideal for research: checking coverage, building URL lists, recovering a handful of files, or feeding your own tooling. For a full website it is only the first step. You still need to choose consistent snapshots, rewrite links to local paths, find assets saved on other dates, and test every page offline. That cleanup is where most DIY projects stall.

If you would rather receive a cleaned, browsable copy, our Wayback website download as HTML service does the checking and link repair for you, and the free archive check tells you how much of the site was saved before you decide anything.

Hitting rate limits while querying or downloading? See how to fix a Wayback Machine 429 error and design a polite download job.

Frequently asked questions

Is the Wayback Machine API free?

Yes. The Availability and CDX APIs are free for public data and need no account. Fair use still applies, so keep request rates modest.

Do I need an API key?

Not for reading public captures. Keys are only involved in other services, such as authenticated Save Page Now requests or restricted data.

What is the difference between the CDX API and the Availability API?

The Availability API returns the single closest capture to a date. The CDX API lists every capture and lets you filter, collapse and page through them.

Can the API download a whole website in one call?

No. It lists captures and serves files one by one. A full download means many requests, which is why downloaders loop over the CDX list with pauses.

Why does the downloaded HTML still link to web.archive.org?

You fetched the normal replay URL. Add id_ after the timestamp to get the original markup without rewritten links.