And because we talked about “How Google crawls your site”, I thought that it will be very useful for you, if I make a list with basic Spidering Tools. These tools will help you to understand how search engines see your site.

Multi tools
The Webconfs.com SEO Toolset - you can find here a spider simulator, a redirect checker, a reciprocal link check and more…
The W3C Validation Service - markup validator, link checker, CSS validator
Webmaster Toolkit - search engine spider simulator, link checker, dns checker and more…

Spider simulators
Search Engine Spider Simulator at Self SEO
Spider/Crawler simulator for websites at oyoy.eu - for different user-agents: Yahoo Slurp, Googlebot, MSN, Archive.org and more…

Link checkers
Free Broken Link Checker at Dead-Links.com

DNS Check
DNSstuff - DNS tools, DNS hosting tests, WHOIS, traceroute, ping, and other network and domain name tools.

Google crawls the websites using spiders. They crawls the net, find sites and add them to his index. Even if you add manually the site using Google Add Url this does not guarantee that your site will be indexed faster.

What can be done? Generate a xml sitemap file of your website. After you generated it, let Google knowing about it - submit it to Google. Your site will be crawled in few days, usually the next day. You will need for sitemap submission a Google account, you can create one in 2 minutes. More information you find at How do I submit a sitemap. Another way of letting Google, Yahoo and Ask to know about your site, is to put your sitemap in the robots.txt file. This is called SE sitemap autodiscovery. They will find it and spider it.

The following is a list of software I found useful for sitemap generating:
Google Webmasters - Sitemaps Third Party Programs and Websites
Very useful for small sites (<500 pages) is XML-Sitemaps.com It is very easy to use, it generates the xml, txt, ror, html sitemap files.

A very useful blog related to Google crawling you can find at Google Webmasters Blog.