And because we talked about “How Google crawls your site”, I thought that it will be very useful for you, if I make a list with basic Spidering Tools. These tools will help you to understand how search engines see your site.

Multi tools
The Webconfs.com SEO Toolset - you can find here a spider simulator, a redirect checker, a reciprocal link check and more…
The W3C Validation Service - markup validator, link checker, CSS validator
Webmaster Toolkit - search engine spider simulator, link checker, dns checker and more…

Spider simulators
Search Engine Spider Simulator at Self SEO
Spider/Crawler simulator for websites at oyoy.eu - for different user-agents: Yahoo Slurp, Googlebot, MSN, Archive.org and more…

Link checkers
Free Broken Link Checker at Dead-Links.com

DNS Check
DNSstuff - DNS tools, DNS hosting tests, WHOIS, traceroute, ping, and other network and domain name tools.