We often get asked by prospective and new clients about Googlebot, such as, "how they know if its been to their site", and so on. Therefore, we've designed this FAQ section to answer some of the more common questions.
1. How do I get listed in Google?2. What's the name of The Google Spider.
3. What's the difference between Deepbot and Freshbot?
4. How do I know if Freshbot and/or Deepbot have visited my site?
5. What's the actual name for Googlebot showing in the logs?
6. How do I know if my site has been spidered by Google?
7. How do I know which spider (Freshbot or Deepbot) has visited my site if my log analysis reports don't tell me?
8. Googlebot has been to my pages what happens then?
9. How do I know if my site has been indexed by Google?
10. It has been a few days since Googlebot visited but I'm still not showing up in the results pages - Why?
11. My site uses dynamic pages. How can I get it indexed in Google?
12. What if my site is unavailable for Googlebot?
13. What other issues will cause my site to not be indexed by Google?
14. Can I block Googlebot from indexing my site?
1. How do I get listed in Google?
Go to this page and request that your site be added. We recommend only submitting the index of a properly linked site and letting Google find your pages on its own. If for some reason you feel that Google will not index your site properly, then we recommend submitting your sitemap page as well.
2. What is the name of The Google Spider?
Google calls its spider "Googlebot". Googlebot comes in two flavors: Deepbot and Freshbot.
3. What's the difference between Deepbot and Freshbot?
Both spiders have specific tasks, as you may have guessed by their names. Freshbot is the spider you hope to see more often. It looks for fresh content on a site. It is not uncommon for Freshbot to visit a site many times a day. Deepbot is responsible for the really deep crawling. Generally when you see search results change, it is because Deepbot has been active. Deepbot is responsible for thoroughly crawling a site and attempting to build a complete picture, or matrix, of the site; how the site is interlinked and how its navigation affects its usability. Deepbot uses data gathered by Freshbot as well as its own results to build a picture of your site.
4. How do I know if Freshbot and/or Deepbot have visited my site?
Generally you can tell by the spider's IP address. Deepbot uses IPs that start with 216, while Freshbot uses IPs that start with 64. In other words, a Deepbot IP would resemble 216.239.45.4 while a Freshbot IP could include 64.208.32.4.
5. What's the actual name for Googlebot showing in the logs?
Googlebot/2.1 (+http://www.googlebot.com/bot.html). This appears for both Freshbot and Deepbot.
6. How do I know if my site has been spidered by Google?
The easiest way is to do a search for Googlebot (http://www.googlebot.com/bot.html) in your logfile. It may not appear in the "spiders" section of your log analysis tool, however, as it tends to emulate a browser instead. Therefore, if you are using a log analysis tool like WebTrends, look in the "browsers" section of the report.
7. How do I know which spider (Freshbot or Deepbot) has visited my site if my log analysis reports doesn't tell me?
You will likely need to look at the raw log files to see which spider visited. If you perform a search in the file for "Googlebot" then look for the IP address. It will be in the ranges listed above.
8. Googlebot has been to my pages, so what happens next?
Generally, you can expect your site to start showing up in the search results in a short time. Google makes no guarantees when or even if your site will be included in the index.
9. How do I know if my site has been indexed by Google?
If you have noticed the Googlebot appearing in your log files, the easiest way to see if you have been indexed by Google is to perform a search for your site. Simply search for your site (i.e. www.mysite.com or mysite.com) and see if your pages show up. Upon performing a search you should see something like this:
The above example shows that our site, Searchengineposition.com is indexed in Google because Google is showing our current title, and some page text as well as other information about the site.
10. It has been a few days since Googlebot visited but I'm still not showing up in the results pages - Why?
Generally, it takes more than a few days to show up in the index. We recommend waiting at least 1 month as Google usually regularly updates its index in this timeframe. If it has been more than two months, there may be other issues which affect your site's ability to be indexed. As mentioned above, Google makes no guarantees when, or even if, your site will show up in the search results pages. Go to this page on the Google site to see reasons why your site may not be indexed. Aside from Google deciding not to list your site, there are other issues which could have affected your being indexed. Read more about them here.
11. My site uses dynamic pages. How can I get it indexed in Google?
Google does index dynamic sites on its own. The problem will be that it won't rank them highly. If all you are concerned with is getting into the index, then you are ok. If you want to rank well for key phrases though, you should consider alternatives other than a dynamic URL system to display your site.
12. What if my site is unavailable when Googlebot visits?
Generally, both Deepbot and Freshbot will make repeated attempts to access your site before moving on. Therefore, it's recommended that your site be available for a majority of the time. If you were indexed in Google, then were removed because your site was unavailable, we recommend waiting at least a month to see if you get reindexed. Many times Google will remove an inaccessible site, to keep its results relevant, then will reinstate the site when the site is available again.
13. What other issues will cause my site to not be indexed by Google?
There can be many things, aside from your site not being available, which could cause Googlebot to exclude your site from the current crawl and index. There are many server issues which could affect ranking as well as design issues and other issues which make it difficult to index the site.
14. Can I block Googlebot from indexing my site?
While we do not recommend this in any situation, unless you thoroughly understand how to write this file, you may feel the need to block all or part of your site from spiders. Through the use of a file called robots.txt you can exclude specific files and folders from being included in the index. You can even block your whole site from being indexed, therefore, you should only employ this file when you are sure you have it configured properly. For more information on proper configuration of a robots.txt file, click here.
Posts
No comments:
Post a Comment