One tool that you can use to "see what the search engines see" is to use a search engine
spider simulator, such as the one linked to below. If you cannot see all of the links to your site pages after performing a simulated crawl, then you will need to take a look at your code to assess what may be getting in the way of effective search engine crawling.
Search Engine Spider Simulator:
A great way to build a search engine-friendly road map for search engine spiders to follow and index site pages is to create Site Map with simple text links to all of the pages on your site. A Site Map also has the added usability benefit of providing visitors with an overall snapshot of all site pages.
Link Popularity: The Importance of Inbound Links
Another way that search engines find your site pages is by following links to your site from
other external sites (provided that those sites themselves are listed in a given engine). Having such links to your site not only provides search engines with additional opportunity to find your pages, but also provides increased visibility for your site by putting it in front of visitors on another site. Many top search engines, such as Google, will factor in the number of sites linking to yours in determining its results for a particular search query. This is known as "link popularity". One way to think about link popularity is that each external link to your site counts as a "vote" for your site. So, the more links you have pointing at you the better, right? Well, not necessarily. Because search engines also know how to count the link popularity of the sites linking to yours, a single link from a popular site will weigh more heavily than many links from obscure unpopular sites. When it comes to getting links, quality over quantity is the way to go.
All things being equal, of two sites with content of comparable relevance and structure, the site with higher link popularity will get more prominent placement in search engine results.
Who's Linking At You?
If you want to research pages that currently link to your site, Google supports a neat little trick that lets you do just that. If you go to google.com and type in your URL proceeded by "link:" (without the quotes "" e.g. link:www.your-url.com) the results will give you a list of all pages listed within google that link to your site. You can also use this method to research sites that linking to organizations similar to yours "" a site that links to organizations like yours, or even a direct competitor, may be a good lead to follow-up with in requesting a link of your own.
A great way to build a search engine-friendly road map for search engine spiders to follow and index site pages is to create Site Map with simple text links to all of the pages on your site.
Originally posted 2007-03-08 23:36:15.