Subscribe to Marketing Jive
 Subscribe to our feed.
Marketing-Jive, formerly SEO-Space, was established in 2006 and since then we have noticed significant increases in both traffic and feed subscribers. If you want to promote your business to thousands of visitors who understand digital marketing, you’ve come to the right place. Sign up and start receiving qualified leads right now. Your ad will be visible on every unique page on our blog.
Glossary of SEO Terms
  • SEO Terms A-C
  • SEO Terms D-F
  • SEO Terms G-I
  • SEO Terms J-L
  • SEO Terms M-O
  • SEO Terms P-S
  • SEO Terms T-V
  • SEO Terms W-Z
  • Enquiro's Online Marketing Glossary
Search Engine Market Share (US)
Organic Search / SEO Tips
3 Ways to Help Google Crawl Your Website
Sunday, August 07, 2011
You know SEO has been around long enough that most webmasters should have a good idea about how effective their site is being crawled by the search engines. Of course there are those site owners who continue to tinker with their sites in order to refresh their look and feel or to try and gain better placements within the search results. The fact is, is that it is all about relevancy (but that is a topic for a different day).

If you are trying to market your site on the vastness that is the Web, one of the first things you will want to put in place is the ability for the search engines to find and crawl your website's pages. If the search engines cannot find your content, how if the *bleep* do you expect to rank for a given keyword? You can optimize all you want, but if your site cannot be easily crawled you are not going to have much success from an SEO and search traffic perspective. So then what measures can you take to ensure that the search engines crawl your content?

Well first and foremost, you need to have an idea of the pages that make up your site. As I wrote last week, the importance of having an inventory of your site pages is extremely important.  How can you expect Google to crawl all of your site's important pages when you are not even sure how many important site pages that you currently have?  So take inventory of your site pages, even if you are a large e-commerce site, you should keep an inventory of your site URLs handy.  At this point I think that it is important to make reference to what I thought was another excellent post from ex-Google Vanessa Fox with her piece on the Fetch Googlebot functionality that can now be used to submit site URLs.  Vanessa reminds us how Google finds your site pages through their discovery process which mainly consists of the following:
  1. Through links
  2. RSS feeds
Google continues to have the best discovery of pages for their index... perhaps too good if you consider some of the issues that they are having with their updated roll out of Caffeine in which Google's index grew exponentially. Regardless, Google has a massive amount of pages in their index. The question is do they have all of your pages indexed (at least the ones you want indexed) and if not how can you ensure that your pages get indexed in Google?

Let's now discuss the purpose of this piece and that is to talk about ways of helping Google crawl your website.

3 Ways to Help Google Crawl your Website

Googlebot crawls the Web via links.  If by chance your content is not linked to and is not well interlinked itself, Google may be missing key pieces of your site in their index.  However there are ways that you can help Google find your content.  The following methods are three basic ways to help get Google to crawl your site.
  1. XML Sitemaps - this is probably a preferred method due to the control that you have, although Google may still not crawl all URLs submitted, XML sitemaps definitely point Googlebot in the right direction.  As Vanessa points out in here article, "The XML Sitemaps protocol enables you to submit a complete list of URLs to Google  and Bing. The search engines don’t guarantee that they’ll crawl every URL submitted, but they do feed this list into their crawl scheduling system."  More on Google and XML sitemaps here
  2. Submit your site to Google - have a new site? You might want to kick it old school and submit your URL to Google.  You will need a Google account and you can go to https://www.google.com/webmasters/tools/submit-url?pli=1 and follow the instructions for submitting your URL.
  3. Through GWT's Fetch as Googlebot -ah yes this is new as of early August.  The Fetch as Googlebot option allows you to request indexing consideration for that page after you fetch it by clicking Submit to index.  This new feature is ideal for situations when you launch a new set of pages on your site or have major updates.  Using this option, you can submit up to 50 URLs a week.  Again see Vanessa's post for additional details.
If you are looking for mass submission of URLs to Google, you will want to create an XML sitemap.  You will need multiple sitemaps if you have over 50,000 or more URLs.  Remember that the real purpose of XML sitemaps is not to submit URLs that Google already have in their index, it is designed for pages that Google may be having difficulty in finding.  Of course if Google or the other search engines are having trouble finding your content, you may have larger concerns:
  • have you inadvertently blocked the engines from finding certain pages?
  • have you properly linked to new site pages?
  • is your site being penalized?
    At the end of the day, anything that you can do to help the engines find your content is always a good thing.  If you are a regular user of Google Webmaster Tools, I recommend that you try the Fetch as Googlebot functionality to try and get new pages crawled and indexed by Google.  The three options above can be used by any siteowner or Webmaster, but there are certain times when one of the options is more purposeful over the others.

    Share |

    Labels: ,

    posted by Jody @ 3:12 PM  
      Post a Comment
      << Home
      Top B2B Blogs   
      Invesp landing page optimization
      About Me
      Name: Jody
      Home: Kelowna, BC, Canada
      About Me: SEO guy by day, family man 24/7.
      Previous Posts
      Marketing Jive Vault of Posts
      Online Marketing Resources
      • Optimizing for Blended Search
      • Search Engine Guide
      • WebProNews Canada
      • Official Google Blog
      • Yahoo Search Blog
      • Search Engine Watch
      • 100% Organic
      • Global Thoughtz
      • B2B Marketing Blogs
      • Silicon Valley Gateway
      • Guy Kawasaki
      • Church of the Customer Blog
      • Marketo's Big List of B2B Blogs
      Blogs We Like
      Hockey Fanatic
      30-Five: Parenting Tips
      Ask.com Blog
      Comparison Engines
      Matt Cutts

      Yahoo Search Blog

      Add to Technorati Favorites

      Marketing Jive Home


      Subscribe | | Advertise | Site Map

      Add to GoogleAdd to My Yahoo!Add to BloglinesAdd to NetvibesAdd to Windows Live