Websites Blocking Domaincrawler 1.0

Site Stats Database currently tracks 1,189 websites that block Domaincrawler 1.0. We detect this by analyzing each site's homepage, response headers, and DNS, then enrich every record with traffic and authority data.

By number of sites, Domaincrawler 1.0 is #289 of 10,135 blocked crawlers & bots that Site Stats Database tracks.

The most common topics among them are Athletics, University, and College, and they are most common in United States. The breakdowns above show the full split, based on the 1,189 we track.

They tend to have stronger authority than the typical website: the median Domaincrawler 1.0 site has an Ahrefs Domain Rating of 44, higher than the 28 median across all sites Site Stats Database tracks.

Over the last 2 months, the number we track grew 177% and their combined traffic rose 6.7%, based on Site Stats Database's month-by-month snapshots of this group.

Traffic estimates, organic keywords, and authority come from DataForSEO; Domain Rating comes from Ahrefs' public endpoint; and the topic, market, and tech detections come from Site Stats Database's own crawler, which inspects each site's HTML, headers, DNS, and TLS certificate.

At least monthly. Traffic and rankings refresh every month, and detections refresh as sites are re-crawled, so this page always reflects the current 1,189 websites that block Domaincrawler 1.0.

Yes. The complete, filterable list of all 1,189 websites that block Domaincrawler 1.0, enriched with traffic history, authority, keywords, and the full tech stack, is available with a Site Stats Database plan.