1. Size of your site - First, you need to have some idea of just how many pages your site(s) consists of.  For smaller sites (<100K pages) this is fairly easy to get a grasp of.  There are various sitemap generator tools that you can use to compile a list of your site's pages.  Quite honestly it amazes me about how many site owners do not actually have an understanding about the actual size of their web sites.  I mean I understand that larger sites can have tons of dynamically generated pages, but you should still have some assemblance of the actual size of your site.  Sitemaps are important not only for your visitors but also for search engine crawlers. 
   
   I recommend keeping a periodic inventory of your site pages.  This is especially helpful if you are planning a site redesign as you will want to have an idea of the sheer volume of your site pages.  It is also great to monitor your site as it grows with fresh content that is added.  Consider keeping a timeline of page size of your site.  How many of you can honestly say that you know how many pages  existed on your site say a year ago?  Two years?  Five years ago?  Keep tabs of this in a spreadsheet, it is not a difficult thing to do.  This data can be very valuable in the future.
   
2. Understand how to monitor the # of pages that you have indexed in the search engines - there are a number of tools that can be used to accomplish the best.  We're gonna kick it old school and mention a couple of ways that you can monitor the number of pages indexed in the major search engines:
   
   • Use the "site: command".  This works well in Google, Bing and to a lesser extent Yahoo.  Let's say that I wanted to see how many pages that I had indexed in Bing.  I would simply go to Bing and type in the following query:  "site:www.marketing-jive.com".  Now there are variations of this command that you can use.  If you have sub-domains, you'll want to use the query "site:www.marketing-jive.com" sans the www.  Here's what the results for Marketing Jive look like in Bing:
     
     
     
     
     
     
     Bing is reporting that I have roughly 1,600 pages indexed within Bing.  If I perform the same query in Google, it tells me that I have about 432 pages indexed.  In Yahoo, it states that I have 1,671 pages indexed.  What's interesting about this is that while not exact, Yahoo and Bing are more accurate than what Google is reporting.  News Flash:  Each of the major search engine crawlers have different manners of crawling and indexing web content.  You should expect to see a variation in the data... do not be alarmed by this.  Furthermore, should I panic because Google is reporting a lower number of pages being indexed than say Bing?  Perhaps, but I know that they are only showing a sample of what I actually have indexed. I can use a second method to evaluate the number of pages that I have indexed in Google...
     
     
     
   • 
     
   • Use Google Webmaster Tools - this is a great tool for site owners and webmasters.  You can create an XML sitemap and submit it to Google.  From here you have access to a number of key areas including crawling and indexing issues.  We should point out however that via Google Webmaster Tools, Google often reports that only a fraction of the pages being submitted via XML sitemaps are currently indexed in Google. This is normal, but it can also be a sign that Google may be having issues with crawling and indexing of your site's pages.  The point here is to be careful of taking Google's word as the Gospel here.  Cross reference with the "site:" command to get a more accurate reflection.  Typically the numbers that you get from Yahoo and Bing are a more accurate assessment of the try number of pages that you have indexed.  Again, Google tends to show but a sample.
   
   
3.  Spot Checks are Important - realize that you should be performing regular spot checks in the engines and especially Google to see if there are certain sections of your site that are having indexing issues.  An easy way to perform a spot check for a given section of your site is to use the "site:" command.  Let's say for example I wanted to see how many pages Google has indexed for my 2010 archives.  I would simply type the following command into Google:
   site:www.marketing-jive.com inurl:2010

1. Cross reference with the other engines
2. Examine your Google Webmaster Tools data
3. Check your robots.txt file
4. Ensure your XML sitemaps are up to date and have no errors
5. Spot check sections of your site using the "site:" command
6. Monitor your meta data - have you inadvertently placed a "no index and/or no follow" on a certain section of your site?
7. Keep an eye on crawling and indexing trends

Labels: google webmaster tools, search engined indexing