How to Find All Pages on a Website (6 Ways That Work)
There is no single button that dumps every URL on a website. But there are six methods that, used together, get you very close, and each one is free. Some read the list the site publishes about itself, some rebuild the list by crawling, and one pulls the list out of an archive. This guide walks each in order, from the fastest to the most thorough, and is about discovering the pages that exist on a site, not the pages you personally visited. If that is what you actually want, skip to the last section.
Start with the site: operator, because it takes ten seconds and needs no tools.
1. The site: search operator
Every major search engine understands site:, a filter that restricts results to one domain.
- In Google, Bing, or DuckDuckGo, search for
site:example.com. - Add a path to narrow it:
site:example.com/bloglists only pages under/blog. - Add a keyword to filter further:
site:example.com pricing.
The result count near the top is a rough estimate of how many pages the engine has indexed for that domain. Page through the results to see the actual URLs.
The limit to understand: site: only shows pages the search engine has crawled and indexed. New pages, pages blocked by robots.txt, and pages with no inbound links may never appear. So site: is a fast floor, not a complete list.
2. The site's sitemap.xml
Most sites publish a sitemap, an XML file that lists the URLs the owner wants search engines to find. It is the closest thing to an official page list.
- Try
example.com/sitemap.xmlandexample.com/sitemap_index.xmldirectly in your browser. - Large sites use a sitemap index that links to several child sitemaps (posts, pages, products). Open each child to see its URLs.
- A sitemap entry often includes a
lastmoddate, so you can also see when each page last changed.
Sitemaps are authoritative but not exhaustive: they include what the owner chose to list. Orphaned or intentionally unlisted pages will not be there. Still, for a well-run site this is the single best source.
3. robots.txt
Before crawling, check example.com/robots.txt. This small text file does two useful things.
- It usually names the sitemaps with one or more
Sitemap:lines, which is often how you find sitemaps that are not at the default path. - Its
Disallow:rules reveal directories the owner does not want crawled, which is itself a hint about the site's structure (admin areas, search pages, staging paths).
robots.txt will not list every page, but it points you to the sitemaps in method 2 and sketches the shape of the site.
4. The Wayback Machine and its CDX API
The Internet Archive has been recording public pages for decades, and it will hand you a bulk list of every URL it has ever captured for a domain.
- For a quick look, browse
web.archive.org/web/*/example.com/*. - For a full machine-readable list, use the CDX API. Open this URL, replacing the domain:
http://web.archive.org/cdx/search/cdx?url=example.com*&output=text&fl=original&collapse=urlkey
The collapse=urlkey parameter removes duplicate captures so you get one row per unique URL. This is the most powerful method for finding old or removed pages, because it includes URLs that no longer exist on the live site.
The tradeoff: the archive only knows pages it managed to capture, and it can include stale URLs that returned errors. Treat its output as a superset to verify, not gospel.
5. A free crawler
A crawler starts at the homepage, follows every link it finds, and repeats until it has walked the whole site. This discovers pages that are linked but missing from the sitemap.
- Screaming Frog's SEO Spider crawls up to 500 URLs on its free tier, which covers most small and mid-size sites.
- Lighter open-source options exist if you are comfortable on the command line, and a simple
wget --spider --recursivewill enumerate linked URLs without downloading the content.
A crawler finds only what is reachable by following links from where it starts, so pages with no inbound links (the classic "orphan" pages) still slip through. Run it alongside the sitemap for the best coverage.
6. On-site search and navigation
Finally, the site itself often exposes its own contents.
- Use the site's internal search box with a broad term, or an empty query where the software allows it, to list many pages at once.
- Read the footer and the HTML/CMS navigation, which frequently link to category and archive pages that roll up everything.
- If you run the site, your CMS (WordPress, Ghost, and others) lists every page and post in its admin, which is the only truly complete source.
Finding the pages you already visited on a site
The six methods above answer "what pages exist on this website." A different and common question is "what pages did I read on this website," and search engines cannot answer that, because your personal browsing is not in their index.
That is a browser-history problem. If you are trying to retrace the pages you actually visited on one site, see the companion guide, how to search all pages of a website you visited. In short, Chrome's chrome://history can filter to a domain, and a local tool like TraceMind indexes the full text of the pages you visited so you can search them by content, not just by title, entirely on your own device.
Putting it together
For a fast estimate, use site:. For the owner's intended list, read sitemap.xml (found via robots.txt). For historical and removed pages, query the Wayback CDX API. For linked pages the sitemap missed, run a crawler. Combine two or three of these and you will have a list that is complete enough for an audit, a migration, or a research sweep.
FAQ
How do I see every page on a website for free
Combine free methods: search site:example.com, open example.com/sitemap.xml (its location is usually listed in example.com/robots.txt), and query the Wayback Machine CDX API for historical URLs. For linked pages missing from the sitemap, run a free crawler such as Screaming Frog's free tier (up to 500 URLs). No paid tool is required for most sites.
What is the site: search operator
site: restricts a search to one domain. Searching site:example.com in Google, Bing, or DuckDuckGo returns only pages that engine has indexed on that domain, and you can add a path (site:example.com/blog) or a keyword to narrow it. It is the fastest way to estimate how many pages a site has, but it only shows indexed pages.
Where is a website's sitemap
Most sitemaps live at example.com/sitemap.xml or example.com/sitemap_index.xml. If neither works, open example.com/robots.txt, which usually names the sitemap with a Sitemap: line. Large sites split their sitemap into an index that links to several child files.
How can I find pages that were deleted from a website
Use the Internet Archive's Wayback Machine. Its CDX API returns every URL it ever captured for a domain, including pages that no longer exist: open http://web.archive.org/cdx/search/cdx?url=example.com*&output=text&fl=original&collapse=urlkey. This is the most reliable way to recover removed or historical URLs.
How do I find the pages I personally visited on a site
Search engines cannot help, because your browsing is not in their index. Use your browser history: chrome://history filters by domain, and a local extension like TraceMind indexes the full text of pages you visited so you can search them by content on your own device. See the guide on how to search all pages of a website you visited.