How to Find All URLs on a Website (4 Methods)

On this page, 10 sections

To get all page URLs from a website, read its robots.txt for a Sitemap: line, open that file and copy every <loc> value. If there is no sitemap, follow links with a crawler. On WordPress, the REST API lists every published post. No single method is complete, so run two and compare them.

I did that comparison on my own site on 11 October 2026. At first the sitemap listed 400 post URLs and the REST API listed 779 published posts, so 379 posts looked missing. 163 of them were hidden by a stale sitemap index that I then fixed. The other 216 redirect elsewhere and are rightly left out. The sitemap alone would never have told me any of that.

This post is the general version. The sitemap method with IMPORTXML in Google Sheets is one route inside it. Below are four methods, what each one misses, and a script that diffs two of them.

The four methods and what each one misses

Method	What it returns	What it can miss
Sitemap	URLs the site owner chose to list	Anything left out, and anything a stale cached index hides. On my site: 379 of 779 posts at first, 216 after the fix.
Crawler	URLs reachable by following links	Pages with no inbound link, and anything blocked by robots.txt
WordPress REST API	Every published item of one content type, 100 per request	Other CMSs. Other content types unless you query each. May be disallowed in robots.txt.
`site:` search	Pages Google shows for that site	The docs do not say the list is complete, and I found no export

I read the vendor pages for each on 11 October 2026. Only the sitemap and REST rows are backed by a test of my own. The crawler row is logic: a page that nothing links to cannot be found by following links.

Method 1: read the sitemap

Open https://example.com/robots.txt and look for lines starting Sitemap:. Google’s sitemap guide says you can list several, with no limit on how many.
No such line? Try /sitemap.xml and /sitemap_index.xml. These are common locations, not a rule.
If the file contains <sitemapindex>, it is a list of other sitemaps. Open each child and collect its <loc> values.
Count what you collected. A single sitemap file is limited to 50,000 URLs or 50MB uncompressed, and an index file may list up to 50,000 sitemaps (sitemaps.org protocol, read 11 October 2026).

My site’s robots.txt has one Sitemap: line pointing at /sitemap_index.xml. That index now lists five child sitemaps: three for posts (200, 200 and 163 URLs), one for pages (6) and one for categories (24), 593 URLs in total. When I first fetched it, the index listed only four: the first two post sitemaps, pages and categories, 430 URLs.

The catch is in the name. A sitemap is a list the site owner wants indexed. Nothing forces it to be the whole site, and mine plainly was not.

Pulling a sitemap into Google Sheets

The formula in the earlier post on this site reads every <loc> from a sitemap. Google’s IMPORTXML page documents the syntax as IMPORTXML(url, xpath_query, locale).

For a sitemap index you run it once on the index, then once per child. I ran the same XPath in xmllint against my real sitemap index and it returned 4 child sitemaps, which matches the file. I could not open a Sheet from this session, so I did not run the formula inside Sheets.

Method 2: crawl it

A crawler starts at the homepage and follows every internal link. It finds what visitors can reach, which is a different set from what the owner listed. Compare the two and the gap is your orphan pages.

Screaming Frog’s pricing page says its free SEO Spider crawls up to 500 URLs, and a licence lifts that (read 11 October 2026). On a site my size that cap would stop a crawl well short.

I did not run a crawler for this post, so I have no count to give you. If you point one at someone else’s site, keep to the limits I use: read robots.txt first, no more than 2 requests at a time per domain, 1.5 to 3 seconds between requests, and stop at the first 429 or 403. Only crawl sites you own or have permission to crawl.

Method 3: ask the WordPress REST API

On WordPress the posts endpoint returns published posts as JSON. The handbook says per_page is capped at 100, and each response carries X-WP-Total (all records) and X-WP-TotalPages headers (WordPress developer handbook, read 11 October 2026).

curl -sI "https://example.com/wp-json/wp/v2/posts?per_page=1" | grep -i x-wp-total
curl -s "https://example.com/wp-json/wp/v2/posts?per_page=100&page=1&_fields=link"

Pages live at /wp-json/wp/v2/pages. Custom post types have their own route. /wp-json/wp/v2/types lists each type with its rest_base, and I read that on my own site to confirm the posts route is posts.

One thing to check before you use it on a site that is not yours: my robots.txt contains Disallow: /wp-json/ for all user agents. A well-behaved crawler would skip the REST API on my site. I ran it there because it is mine. On anyone else’s, read their robots.txt first.

Method 4: the site: operator

Google’s search help page lists site: as the operator for results from one site, with site:youtube.com cat videos as its example (read 11 October 2026). The page does not say the results are a complete list of indexed pages. So treat it as a spot check, not an inventory.

Search Console is nearer to an inventory. Google’s Page indexing report help says the View data about indexed pages link shows an example list of up to 1,000 indexed URLs (read 11 October 2026). The same page says the list is not guaranteed to show all URLs, so treat it as a sample, not an inventory.

What I found when I compared two methods

I wrote a short Python script that reads the sitemap named in robots.txt, walks any index, pages through the REST API and prints what is in one and not the other.

I compiled it and ran it against my own site on 11 October 2026. My first run printed 400 post URLs in the sitemap against 779 in the REST API. After I fixed the cause, described below, this is the output:

sitemap URLs: 593
REST posts:   779
in REST, not in sitemap: 216

All 563 post URLs in the sitemap were also in the REST list (779 minus 216 is 563). The 30 other sitemap entries are the 6 page URLs and 24 category URLs. So the sitemap was accurate about what it listed. The question was why the first run saw only 400 posts.

The cause was the sitemap index. It was a stale copy generated on 5 September 2026 that listed two post sitemaps. A third one, post-sitemap3.xml, was live and returned 200 with 163 post URLs, but the index never pointed to it.

A crawler that follows the index would never have found those 163 posts. I deleted the cached index file my SEO plugin had written, purged the server’s page cache, and the index now lists three post sitemaps and 563 posts.

That left 216 posts in the REST API and not in the sitemap. I requested three of them. All three returned a 301 redirect, so they are not live pages at those URLs, and a sitemap should not list them. I did not request the other 213, so I cannot say every one of the 216 redirects.

The script

#!/usr/bin/env python3
"""Compare a site's sitemap URLs with its WordPress REST post URLs.
Only run this on a site you own or have permission to crawl."""
import re, sys, time, urllib.request, urllib.robotparser

site = sys.argv[1].rstrip("/")
own = "--own-site" in sys.argv   # needed when robots.txt disallows /wp-json/
DELAY = 2.0
UA = "Mozilla/5.0 (url-audit script)"

def get(url):
    time.sleep(DELAY)
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    return urllib.request.urlopen(req, timeout=40)

rp = urllib.robotparser.RobotFileParser(site + "/robots.txt")
rp.read()
if not rp.can_fetch("*", site + "/wp-json/wp/v2/posts") and not own:
    sys.exit("robots.txt disallows /wp-json/. Re-run with --own-site only if you own this site.")

def sitemap_urls(url):
    body = get(url).read().decode("utf-8", "replace")
    locs = re.findall(r"<loc>\s*([^<\s]+)\s*</loc>", body)
    if "<sitemapindex" in body:
        return [u for child in locs for u in sitemap_urls(child)]
    return locs

start = (rp.site_maps() or [site + "/sitemap.xml"])[0]
in_sitemap = set(sitemap_urls(start))

in_rest, page, pages = set(), 1, 1
while page <= pages:
    r = get(f"{site}/wp-json/wp/v2/posts?per_page=100&page={page}&_fields=link")
    pages = int(r.headers.get("X-WP-TotalPages", 1))
    in_rest |= set(re.findall(r'"link":"([^"]+)"', r.read().decode().replace("\\/", "/")))
    page += 1

missing = sorted(in_rest - in_sitemap)
print("sitemap URLs:", len(in_sitemap))
print("REST posts:  ", len(in_rest))
print("in REST, not in sitemap:", len(missing))
print(*missing[:5], sep="\n")

It sleeps 2 seconds between requests, refuses to touch the REST API when robots.txt disallows it unless you pass --own-site, and only reads. Run it with python3 urls_diff.py https://yoursite.com.

Which method to use

You want a quick list for a Sheet: the sitemap, then diff it against something.
You own a WordPress site and need the full list: the REST API, per content type.
It is not WordPress, or you need what visitors can reach: a crawler, within the limits above.
You only need to check one thing exists: site:, as a spot check.

Once you have the list, the same sheet can pull page data. Getting data from a website into Google Sheets covers which import function fits which page.

Questions people ask

Can I get all URLs from a website without a sitemap?

Yes, by crawling or, on WordPress, through the REST API. Both have limits: a crawler cannot reach unlinked pages, and the REST API may be disallowed in robots.txt or switched off.

How many URLs can a sitemap hold?

50,000 URLs or 50MB uncompressed per file, and an index can list 50,000 sitemaps (sitemaps.org protocol, read 11 October 2026).

Does a sitemap list every page on my site?

Not necessarily. The first time I checked, on 11 October 2026, mine listed 400 of 779 published posts. A stale index hid 163 of the gap and 216 posts redirect elsewhere, so the sitemap was right once the index was fixed.

Do this today

Run the script on a site you own. If the REST count and the sitemap count differ by more than a handful, find out why before you use the sitemap for an audit or a migration.

Resources

Sitemaps XML format, sitemaps.org, read 11 October 2026
Build and submit a sitemap, Google Search Central, read 11 October 2026
REST API pagination, WordPress developer handbook, read 11 October 2026
Refine Google searches, Google Search Help, read 11 October 2026
Page indexing report, Search Console Help, read 11 October 2026
SEO Spider pricing, Screaming Frog, read 11 October 2026
IMPORTXML, Google Docs Editors Help, read 11 October 2026

How to Find All URLs on a Website: 4 Methods and What Each Misses