You've created great content for your site and are ready to provide value to your users. There's just one problem: your content isn't being indexed properly, which means searchers can't find your web pages with a Google search! 

Creating amazing SEO copy that users (and search engines) love is pointless if that content isn't landing in the search engines' indexes. Since SEOs where many hats, I'm going to assume that you have a limited amount of time for finding and dealing with this problem, so I'm going to show you the three most common problems with search engine indexing so you can quickly tackle low-hanging fruit opportunities. 

Here are a few immediate things to check on your site ... 

1. Paginated Posts, Categories, or Search Results

Let's assume you manage an ecommerce website that has a category, of say, Avatar Toys. Now lets say that the site has 250 products that are related to Avatar and the site shows 25 products per page. That means that there are 10 pages of product results.

Because you were thinking about your categories in the abstract, you didn’t really put much thought into optimizing for anything other than the category as a whole. You might have a title that looks like this:

Avatar Toys, Action Figures, Games and Toys from The Movie Avatar

You've correctly included the target keyword in the copy, but the problem is that your CMS thinks the same way you do: It doesn’t differentiate page 1 from pages 2-n with regard to the titles and meta data it has stored. So, all 10 pages will have the same exact title and meta data.

While the search engines will usually differentiate the pages by rewriting page titles for you (more on that in my page title rewrite analysis walkthrough), you are not going to see great rankings for anything when you have 10 pages with the exact same title.

Depending on how generic (or absent) the on-page optimization is, you may end up with several pages of product listings missing entirely from the index. Even assuming you have additional content on the main page and/or unique headings, search engines are still going to treat you poorly in many cases.

Quick Fix

Have the page number dynamically appended to the end of pages 2-N. In this case, page 1 would say

Avatar Toys, Action Figures, Games from The Movie Avatar

Each subsequent page would say:

Avatar Toys, Action Figures, Games from The Movie Avatar — Page 2 of N

Avatar Toys, Action Figures, Games from The Movie Avatar — Page 3 of N

Depending on your resources and platform, the programming required may take a 2-8 hours but in many cases this update is just a question of changing some settings in your admin.

Recommended Reading: Pagination vs. Infinite Scroll: What's the Difference?

2. Duplicate Navigation Paths

If your site uses merchandising software like Omniture Merchandising, directory software, or internal search to give customers what they want, then you may have multiple pages with the same content but different URLs. This happens because even though you created a canonical page for a category/topic/etc., your merchandising package isn’t smart enough to always insist that visitors go there.

Lets say you operate a travel site for Bermuda vacations and the site has a page for “Paget Resorts” with a URL like this:

http://www.bermudatravelfun.com/paget

But if someone goes to your site and uses the search box instead of your standard navigation, they get a URL like this:

http://www.bermudatravelfun.com/search?q=paget

You’ll have to look for yourself to know for sure, but often these two pages will have identical content with the only differentiator being titles that you added to your canonical version. Search engines don’t really care which version of the page gets the best treatment in the SERPs ⁠— they just want visitors to find what they are looking for. Because of that ambivalence, if the search result page gets linked to or shared you can find good traffic going to a poorly optimized page while the true page is pushed down the index or completely removed.

Quick Fix

  1. Use noindex meta tags.
  2. Use an .htaccess file, ISAPI_Rewrite, or a native tool of your CMS to create a 301 redirect from top query searches to your optimized pages.
  3. Build additional links to your canonical page to help the search engines out.

3. Bookmarking and Linking Sharing Forms

Many people love to share what they've found on the web, so naturally your site might offer searchers the opportunity to send a link to a friend or bookmark the page they are on. With older sites these sharing tools often require the use of a pop-up form. Unfortunately, each time content is shared a new page is dynamically generated. As you expand your content, you’ll find that each new page you build creates a “shadow page” that you never think of, but that search engines find.

Your URL may look like this:

http://www.yoursite.com/emailfriend.aspx?refurl=http://www.mysite.com/magazinearticle.aspx?id=4294

The biggest problems with these pages are:

  • They have little crawlable content: just a form and some generic text.
  • They don’t link to the rest of the site.
  • They have the same page titles as your real content.

Over time, you’ll end up with a bloated index. I see this a lot with Bing and Yahoo. Google usually cleans them out faster, but they can stick around for years. If you see a lot more pages in your index than you are actually submitting, then check for this.

Quick Fix

  1. Use a robots.txt or .htaccess file to block access to these pages by search engine spiders. Google uses these files to know which pages it is allowed to crawl. 
  2. Include “Email to a Friend” or some other text in page titles to ensure that the email form page and the actual article do not share the same title.
  3. Try ShareThis or AddThis as a solution for sharing content, rather than generating additional pages.

Recommended Reading: SEO Crawlability Issues and How to Find Them

Conclusion

After following these quick fixes, you should start to see your content appearing on the search engine results page. 

This post focuses on just a few things that can have a big impact on your site. It assumes that you have more common duplicate content and spider-friendliness issues under control. If you have the time, then you should definitely conduct a broader analysis of your indexing, but this brief list will let you hit some big index issues quickly and show improvement fast. For further reading on technical SEO, head over to: Technical SEO: Best Practices to Prioritize Your SEO Tasks.

Editor's Note:This post was originally published in March 2010 and has been updated for accuracy.