You've created great content for your site and are ready to provide value to your users. There's just one problem: your content isn't being indexed properly, which means searchers can't find your web pages with a Google search!
Creating amazing SEO copy that users (and search engines) love is pointless if that content isn't landing in the search engines' indexes. Since SEOs where many hats, I'm going to assume that you have a limited amount of time for finding and dealing with this problem, so I'm going to show you the three most common problems with search engine indexing so you can quickly tackle low-hanging fruit opportunities.
Here are a few immediate things to check on your site ...
Let's assume you manage an ecommerce website that has a category, of say, Avatar Toys. Now lets say that the site has 250 products that are related to Avatar and the site shows 25 products per page. That means that there are 10 pages of product results.
Because you were thinking about your categories in the abstract, you didn’t really put much thought into optimizing for anything other than the category as a whole. You might have a title that looks like this:
Avatar Toys, Action Figures, Games and Toys from The Movie Avatar
You've correctly included the target keyword in the copy, but the problem is that your CMS thinks the same way you do: It doesn’t differentiate page 1 from pages 2-n with regard to the titles and meta data it has stored. So, all 10 pages will have the same exact title and meta data.
While the search engines will usually differentiate the pages by rewriting page titles for you (more on that in my page title rewrite analysis walkthrough), you are not going to see great rankings for anything when you have 10 pages with the exact same title.
Depending on how generic (or absent) the on-page optimization is, you may end up with several pages of product listings missing entirely from the index. Even assuming you have additional content on the main page and/or unique headings, search engines are still going to treat you poorly in many cases.
Have the page number dynamically appended to the end of pages 2-N. In this case, page 1 would say
Avatar Toys, Action Figures, Games from The Movie Avatar
Each subsequent page would say:
Avatar Toys, Action Figures, Games from The Movie Avatar — Page 2 of N
Avatar Toys, Action Figures, Games from The Movie Avatar — Page 3 of N
Depending on your resources and platform, the programming required may take a 2-8 hours but in many cases this update is just a question of changing some settings in your admin.
Recommended Reading: Pagination vs. Infinite Scroll: What's the Difference?
If your site uses merchandising software like Omniture Merchandising, directory software, or internal search to give customers what they want, then you may have multiple pages with the same content but different URLs. This happens because even though you created a canonical page for a category/topic/etc., your merchandising package isn’t smart enough to always insist that visitors go there.
Lets say you operate a travel site for Bermuda vacations and the site has a page for “Paget Resorts” with a URL like this:
http://www.bermudatravelfun.com/paget
But if someone goes to your site and uses the search box instead of your standard navigation, they get a URL like this:
http://www.bermudatravelfun.com/search?q=paget
You’ll have to look for yourself to know for sure, but often these two pages will have identical content with the only differentiator being titles that you added to your canonical version. Search engines don’t really care which version of the page gets the best treatment in the SERPs — they just want visitors to find what they are looking for. Because of that ambivalence, if the search result page gets linked to or shared you can find good traffic going to a poorly optimized page while the true page is pushed down the index or completely removed.
Many people love to share what they've found on the web, so naturally your site might offer searchers the opportunity to send a link to a friend or bookmark the page they are on. With older sites these sharing tools often require the use of a pop-up form. Unfortunately, each time content is shared a new page is dynamically generated. As you expand your content, you’ll find that each new page you build creates a “shadow page” that you never think of, but that search engines find.
Your URL may look like this:
http://www.yoursite.com/emailfriend.aspx?refurl=http://www.mysite.com/magazinearticle.aspx?id=4294
The biggest problems with these pages are:
Over time, you’ll end up with a bloated index. I see this a lot with Bing and Yahoo. Google usually cleans them out faster, but they can stick around for years. If you see a lot more pages in your index than you are actually submitting, then check for this.
Recommended Reading: SEO Crawlability Issues and How to Find Them
After following these quick fixes, you should start to see your content appearing on the search engine results page.
This post focuses on just a few things that can have a big impact on your site. It assumes that you have more common duplicate content and spider-friendliness issues under control. If you have the time, then you should definitely conduct a broader analysis of your indexing, but this brief list will let you hit some big index issues quickly and show improvement fast. For further reading on technical SEO, head over to: Technical SEO: Best Practices to Prioritize Your SEO Tasks.
Editor's Note:This post was originally published in March 2010 and has been updated for accuracy.