Talking to a number of clients and others in the industry, there appears to still be some confusion on duplicate content issues. Well yesterday there was a fantastic post over at the Google Webmaster Central Blog on Duplicate Content. The post is so great that I thought that I would reproduce it here in its entirety (creating more duplicate content?)
There are some pretty good tips here from one of the engines themselves. They are emphatically saying that multiple URLs of the same content can dilute link popularity. Well if you have ever wondered why you may not be ranking as high as you should be, start by checking your URL structure. Curious if your site has duplicate content issues? There are various tools that you can use to quickly check to see if your site has duplicate content issues. Well all is said and done (and this is not always easy to accomplish) the best way to deal with duplicate content is to avoid it in the first place.
Google, duplicate content caused by URL parameters, and you
How can URL parameters, like session IDs or tracking IDs, cause duplicate content?
When user and/or tracking information is stored through URL parameters, duplicate content can arise because the same page is accessible through numerous URLs. It's what Adam Lasnik referred to in "Deftly Dealing with Duplicate Content" as "store items shown (and -- worse yet -- linked) via multiple distinct URLs." In the example below, URL parameters create three URLs which access the same product page.
Why should you care?
When search engines crawl identical content through varied URLs, there may be several negative effects:
1. Having multiple URLs can dilute link popularity. For example, in the diagram above, rather than 50 links to your intended display URL, the 50 links may be divided three ways among the three distinct URLs.
2. Search results may display user-unfriendly URLs (long URLs with tracking IDs, session IDs)
* Decreases chances of user selecting the listing
* Offsets branding efforts
How we help users and webmasters with duplicate content
We've designed algorithms to help prevent duplicate content from negatively affecting webmasters and the user experience.
1. When we detect duplicate content, such as through variations caused by URL parameters, we group the duplicate URLs into one cluster.
2. We select what we think is the "best" URL to represent the cluster in search results.
3. We then consolidate properties of the URLs in the cluster, such as link popularity, to the representative URL.
Consolidating properties from duplicates into one representative URL often provides users with more accurate search results.
If you find you have duplicate content as mentioned above, can you help search engines understand your site?
First, no worries, there are many sites on the web that utilize URL parameters and for valid reasons. But yes, you can help reduce potential problems for search engines by:
1. Removing unnecessary URL parameters -- keep the URL as clean as possible.
2. Submitting a Sitemap with the canonical (i.e. representative) version of each URL. While we can't guarantee that our algorithms will display the Sitemap's URL in search results, it's helpful to indicate the canonical preference.
How can you design your site to reduce duplicate content?
Because of the way Google handles duplicate content, webmasters need not be overly concerned with the loss of link popularity or loss of PageRank due to duplication. However, to reduce duplicate content more broadly, we suggest:
1. When tracking visitor information, use 301 redirects to redirect URLs with parameters such as affiliateID, trackingID, etc. to the canonical version.
2. Use a cookie to set the affiliateID and trackingID values.
If you follow this guideline, your webserver logs could appear as:
127.0.0.1 - - [19/Jun/2007:14:40:45 -0700] "GET /product.php?category=gummy-candy&item=swedish-fish&affiliateid=ABCD HTTP/1.1" 301 -
127.0.0.1 - - [19/Jun/2007:14:40:45 -0700] "GET /product.php?item=swedish-fish HTTP/1.1" 200 74
And the session file storing the raw cookie information may look like:
How can we better assist you in the future?
We recently published ideas from SMX Advanced on how search engines can help webmasters with duplicate content. If you have an opinion on the topic, please join our conversation in the Webmaster Help Group (we've already started the thread).
Thanks to the team at Google Webmaster Central for this fantastic information.
- Create unique content pages
- Be smart about planning out your URL structure
- Use proper tracking mechanisms to track your visitors (as opposed to having multiple URLs that goes to the same destination page. or use 301 redirects to redirect URLs with parameters as mentioned above.)
Labels: duplicate content