March 12, 2001

By James Martell

Website: www.Desktop-Publishers.com

Enter search engine themes!
Reading an entire website as one single document.

In an attempt to to serve better search results, most of the major search engines are now employing a new technology that is commonly being referred to as "Search Engine Themes."

Unlike page by page indexing, theme technology is based on reading an entire web site as one single document.

As of 11/2000 Google, Inktomi, AltaVista, Excite, and Fast, are all using themes in their algorithm.  That means that the search results from smaller partners like 4anything, AOL Search, HotBot, GoTo, ICQ, LookSmart, MSN Search, and NBCi (Snap) are also based on themes technology.

This change should not be taken lightly.

The move to themes based indexing promises to change all of the 'norms' we've accepted about search engine optimization.  And as this technology becomes more and more embedded in the search engine algorithms, web site owners may find their once high ranking pages showing up lower and lower in the search results.

The Future of Search Engine Optimization

What is your web site about?  I mean really.  Can you come up with one or two short short sentences that say, "My web site is really about ________________."

In the past the only way for searchers to know what your web site was really about was to visit your site personally or perhaps to view your web site's description in a human edited directory like Yahoo.

That's because, historically, search engines have not categorized web sites.  They've indexed and ranked individual web pages.

Because of abuse, spamming and other factors, this page by page ranking method has often resulted in less than desirable search results.  And as the wealth of information on the Internet has grown, people are no longer willing to accept search results that only identify one page of relevant content.

People now expect search engines to deliver links to entire web sites that are filled with content specific information.

How all this search engine theme stuff works

Search engine theme technology may sound a bit confusing at first, but the concept is really not that difficult.

As new web pages or web sites are added to the search engine database - the search engine retrieves all the information it has on record about the existing domain.

To express this concept a little more clearly let's look at an outline of how this new indexing system differs from the old non-theme technology.

This difference may seem subtle, but depending on your web site theme based indexing can dramatically effect your rank.

Take for example the owners of a hypothetical web site called TheMax.com.  In an effort to save money on domain name registration fees they've hosted multiple sites under different directories and each of the sites are linked to one another.

• TheMax.com - is the primary web site

• TheMax.com/music/ - offers free music downloads

• TheMax.com/isp/ - offers ISP services

• TheMax.com/horses/ - offers info on horses

You can easily see how a theme based spider could be confused.  There is no common topic at TheMax.com.  We can't tell if the web site is about music, ISPs, or horses.  And as a result, a search query for "ISPs" will result in the music web site at TheMax.com being listed lower in the search results.

This is a simplified example but it does make a point.  To put it even more clearly, think about this in the context of a web site such as Sears.com.

Sears sells just about everything.  And one of their top selling products is lawn mowers.  But how is a theme based search engine to know that Sears is one of the largest sellers of lawn mowers in the USA?

Well, the answer is that unless the web developers at Sears give the spider a little help, the search engine can't know that information.  So, on the next page we'll dig a little deeper into exactly how all of this actually works.

Theme Based Spider Food

This is what will happen when a theme based search engine spider crawls your web site.

• Read text of the domain name

• Read text of all sub directories - (i.e. /lawn-mower)

• Read text all web page titles

• Read text of all meta data

• Read text of all headings. <H1> <H2>

• Read text of all outgoing links

• Read text of all ALT text

• Read text of all named anchors

• Read text of all body content

The next order of business is to organize and condense the data.

• Delete all filler words (and, or, the, web)

• Identify recurring keywords

• Stem keywords ('lawns' becomes 'lawn')

• Identify keyword phrases ('lawn' becomes 'lawn mower')

• Weight the keywords using common indexing methods. (For example, page titles and header content are given more weight than body text.)

• Establish the web site theme based on the dominance of keyword and keyword phrases.

That sounds fairly simple and complex at the same time.. All we need to do is keep all of our web sites keyword text intense in every area.  But for many existing web sites, things are not going to be that simple.

If you doubt that, just do a search for 'lawn mower' on AltaVista.  Although Sears is the largest supplier of lawn mowers in the USA and they have one of the most visited web sites in the USA, they do not show up in the top ten search results for that term.  For that matter, they don't show up in the top one hundred.

That's not because the webmasters at Sears have done anything wrong.  Sears actually has a wonderful web site. And as of 11/2000 their inbound link popularity is at 6,884! We can't blame them there.  But because AltaVista is employing theme based technology, the Sears.com web site appears to be diluted with erroneous information.

The spider and search engine are confused.  It can't decide if the Sears.com web site is about tools, lawnmowers, or women's clothing.  And as a result, at least according to AltaVista, the Sears.com web site is not a relevant choice for the search phrase 'lawn mowers.'

Alright...already ...Just tell me what to do already.

• Select a keyword intense domain (i.e. free-isp.com)
  (use the hyphen)

• Name all sub-directories with keyword names (i.e. free-isp.com/free-isp-article-1.html)

• Include a common keyword on all web page titles

• Deliver consistent keyword meta tag data throughout the site

• Use Header Text to deliver consistent theme information

• Assign keyword relevant text links

• Use ALT text on graphic links consistently

• Link to outgoing sites which are on content with yours

• Riddle body text with keyword phrase intense info

• Ask other sites to link to you with keyword relevant links

In other words, the name of the game is consistency, consistency, consistency.