In order for a spider to crawl your website and index it in the search engines effectively, the web address or URL for your webpages should be as simple as possible.
As we’ve discussed in the past, sites with static URLs that are simple are crawled and indexed much more efficiently than those containing dynamic characters and session Identifiers.
Session IDs are most common in ecommerce sites and are embedded in a URL so the website can track their customers from page to page and they are used keep track of items in a customer’s shopping cart. But these IDs cause problems for search engine spiders because they create a large number of links for the spider to crawl. This can create a situation where the search engine indexes essentially the same page over and over. Search engines like Google refer to it as a ‘spider trap’.
Below are a couple of examples of how session IDs can give the appearance of an endless number of pages within a single site. A spider coming to your website may find a page with the following URL:
http://www.yoursite/shop.cgi?id=dkom2354kle03i
This page gets indexed but when the spider returns later to look for new content, it finds the following:
http://www.yoursite/shop.cgi?id=hj545jkf93jf4k
This is actually the same page as before, just with a different special session ID but the spider sees it as a brand new URL. Because of this confusion, search engine spiders are programmed to avoid pages containing these session IDs.
While Google and others are trying to improve their ability to crawl URLs with session IDs, it’s best to avoid them whenever possible. It’s best to avoid them until you absolutely must track what a customer is doing, like when they start adding items to their shopping cart.
It’s also possible to store session IDs in cookies rather than URLs. Changing this may require the expertise of a web programmer though.
The gist of the story is this – the more dynamic variables in a URL, the more difficult it will be for search engines to index your pages. To maximize your position in the search engines, use simple URLs that are easy to locate, crawl and index.