How to Increase Google Crawl Rate

Google is king when it comes to the search engines, which is why its pretty darned important to get them to index your pages quickly & efficiently. So I’ve decided to share some of my experience with you. So you can improve your website crawl rate, get those pages indexed quicker & and increase your website traffic.

1. Frequent Content

This sounds like an obvious one, but from my experience people just don’t do it enough. Google loves content, its text it sees, not much else really. So the more content you have, on a regular basis, it then knows to visit you more frequently, to get the latest stuff. Unless you’re a obsessive blogger, and remember this applies to websites which aren’t blogs too. Aim to add new content, once per week. This should over time, and I mean a few months, get google returning every week to index your new pages.

I do admit this can be tricky, especially if you’re website is a very specific topic, as its not easy to add content that frequent. A few suggestions for this is add news about your product, make sure its unique as this will carry more weight. Another idea is to add a corporate blog, this will give your customers a way to get to know your company personally, and it’s far easier to write generate more content this way, but do not just re-publish press releases…

2. Update Every Page

From the title, this one sounds stupid, but this is a great ‘trick’. Google is stupid, just like the other search engines. A lot of websites use includes as a common element in their pages, like a header or footer. Add a nice strap-line describing your website, and change this frequently and regularly. Using the spoon example you could put something like: “Spoonsite.co.uk provides information on spoons and other cutlery” - when you update this weekly, try and increase or decrease it by at least 30 characters, just so google can then see the page size change, and believe all your pages have been update. It does increase the google crawl rate.

3. Good Linking

Internal linking is as important as external links. If you have good linking, then Google can navigate all your your pages easily, increasing the chances of your pages being indexed. External linking is also important, if someone links to you, when they are crawled, you have a pretty good chance the bot will follow that link through to your site, and hey presto, you’re being crawled again.

4. Just check the obvious

If you’ve go a webmaster tools account, make sure you don’t have your crawl rate set to slower, as this wont be necessary for your website usually, only if you have problems. Also check out your robots.txt - that you haven’t actually blocked some bots from visiting. You will be surprised how many people do this.

5. Another great way to invite Googlebot is to assign unique title and meta-tags for each of your pages.

6. Page load time is another critical factor to be taken care of. If the crawler spends too much time crawling your huge images or PDF files, there will be no time left for it to visit your other pages.

7. It is imperative that your server should always return correct header response.

Ok here are a few tips, enjoy and give them ago.