Posts in the Supplemental Index
This evening I was having a quick look at Google and page rankings. I noticed that man of the pages were not being displayed in Google, yet were first page items in Bing and Yahoo. I haven’t built any links so that’s part of the problem but then I checked the Supplemental Index Tool over here and found that 39% of my web pages were in the supplemental index.
Huh? Having a quick look at the SERP showed that this was duplicate content as I saw over 100 indexed pages at a time when I only had around 30. This was caused by WordPress creating unique links via the menu, the archive and the tag cloud on the website. Google was treating this as duplicate content!
Duplicate content and robots.txt
The best way to protect against this is by using robots.txt to block these items from being indexed.
User-agent: *
Disallow: /wp-admin
Disallow: /wp-content/cache
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
Disallow: /wp-includes
Disallow: /trackback
Disallow: /comments
Disallow: /category
Disallow: /tag
Disallow: /2011/
Disallow: /2012/ # add any other years you require
Edit: oddly enough I only see pages in the supplementary index when I enter www.cmsbloke.com — if I enter cmsbloke.com my index score is 0!
Adding this haccess should help get you out and keeping you out of the Google supplementary index.