Print This Page

Getting Listed In The Major Search Engines
 
After you get your website up and running you will inevitably get a ton of junk mail trying to sell you search engine submissions.  Save your money!!  We have found (from experience) that you can get much better results if you submit to the top search engines by hand. 

Each search engine is a bit different (in many cases a lot different), and there are special things you need to know and watch for when submitting to each of these.  The term "search engine" is often used generically to describe both crawler-based search engines and human-powered directories. These two types of search engines gather their listings in radically different ways.  Here is a brief explanation of the difference:

Crawler-Based Search Engines
 Crawler-based search engines, such as Google, create their listings automatically. They "crawl" or "spider" the web, then people search through what they have found.  If you change your web pages, crawler-based search engines eventually find these changes, and that can affect how you are listed. Page titles, page text and MANY, MANY other elements all play a role. 

Search for anything using your favorite crawler-based search engine. Nearly instantly, the search engine will sort through the millions of pages it knows about and present you with ones that match your topic. The matches will even be ranked, so that the most relevant ones come first. Of course, the search engines don't always get it right. Non-relevant pages make it through, and sometimes it may take a little more digging to find what you are looking for.  But, by and large, crawler-based search engines do an amazing job.

So, how do crawler-based search engines go about determining relevancy, when confronted with hundreds of millions of web pages to sort through?  Each one follows a set of rules, known as an algorithm.  Exactly how a particular search engine's algorithm works is a closely-kept trade secret, and changes all the time.   Every one is different.

Human-Powered Directories
A human-powered directory depends on humans for its listings. You submit a short description to the directory for your entire site, or editors write one for sites they review. A search looks for matches only in the descriptions submitted.  Changing your web pages has no effect on your listing. Things that are useful for improving a listing with a search engine have nothing to do with improving a listing in a directory. The only exception is that a good site, with good content, might be more likely to get reviewed for free than a poor site.

"Hybrid Search Engines" Or Mixed Results
In the web's early days, it used to be that a search engine either presented crawler-based results or human-powered listings. Today, it extremely common for both types of results to be presented. Usually, a hybrid search engine will favor one type of listings over another. For example, MSN Search is more likely to present human-powered listings from LookSmart. However, it does also present crawler-based results (as provided by Inktomi), especially for more obscure queries.

 

Getting Listed!


Submitting to Google
Google has a well-deserved reputation as the top choice for those searching the web.  Arguably it is the most important crawler-based search engine because several major search engines such as Yahoo and AOL Search have their main results "powered" by Google. (Google was originally a Stanford University project by students Larry Page and Sergey Brin called BackRub. By 1998, the name had been changed to Google, and the project jumped off campus and became the private company Google. It remains privately held today.)

The absolutely best way to get listed with Google is to build links to your web site. (This is the best way to get listed for free with all the major crawlers listed on this page.) Crawlers follow links, so if you have good links pointing at your web site, the crawlers are more likely to find and include your pages.  Here's the good news: if you submitted your site to the major directories and got listed with one or more of them, then Google and other crawlers will almost certainly pick up the URL that was listed. This means you may not need to do additional work to get listed with crawlers.  Nevertheless, you may find it helpful to do some link building beyond the directories. 

Aside from link building, Google provides an Add URL page that lets you submit a URL directly to its crawler. There's no guarantee that Google will actually include a URL submitted to it this way, however. Despite this, it makes sense to submit your home page and perhaps one or two other URLs from "inside" your web site via the Add URL page.  You really don't need to submit more than this. The only reason for submitting some of your inside pages is in case there is a problem reaching your home page. This gives Google an alternate route into your site. From whatever page it visits, it will look for links to other pages that you have and perhaps include those. This is true for other crawlers, as well.  If you have a brand new web site, it will probably take about a month before Google lists your web pages.

Tip #1: Google usually indexes sites during the 2nd week and lists new content during the first few days of the next month.  We usually see our new content added on the 2nd or 3rd of the month. A safe bet is to wait for a month to see new pages listed.
 

Tip #2:  Here's a cool trick to find out if you are listed in Google. Type this URL in your browser: http://www.google.com/search? q=YOURDOMAIN+site:WWW.YOURDOMAIN.COM
Google will return to you a complete list of all pages that lie on yourdomain.com that exist within the Google catalog.

You can also find out how many sites that are listed in Google are linking to your site. Here's the trick. Type this URL in the browser and watch the results:  http://www.google.com/search?as_lq=YOURDOMAIN.COM
Google will return to you the number of pages which link to your web site in addition to a listing of each URL.
 



Submitting To MSN Search
Microsoft is known for constantly reworking its software products until they get them right, and MSN Search is a shining example of the company putting that same effort into an online product. In particular, the company has its own team of editors that monitors the most popular searches being performed and then hand-picks sites that are believed to be the most relevant.  After performing a search, "Popular Topics" shown below the search box on the results page are also suggestions built largely by editors to guide you into making a more refined search.

Of course, humans editors can't do everything, so MSN Search also relies on search providers for answers to many of its queries. Usually, it will be human-powered results from the LookSmart.com directory that dominate the page. Unlike when MSN editors are involved, these human-powered results are not hand-picked to match a query. Instead, MSN uses its own search algorithm to sift through all the listings from LookSmart to automatically find answers that are believed to be best.   For more obscure queries, it is crawler-based results from Inktomi that are provided.  Overall, MSN Search provides a blend of human-powered directory information and crawler coverage different from any of the other top choices listed on this page. It's a high quality resource that provides its own unique view of the web and one worth checking.  Getting Listed: You need to be listed with search providers LookSmart and Inktomi, which are described in the "Search Providers" section further below.
Submitting To AllTheWeb.com
 AllTheWeb is an important crawler-based search engine primarily because it powers the primary results for the popular Lycos service. The search engine was also recently purchased by paid listing provider Overture, so its listings could potentially be used by some Overture partners, in the future.  Building links is the best way to get listed for free. However, AllTheWeb also accepts submissions via its Add URL page. As with the other crawlers with Add URL pages, you need only submit your home page and one or two inside pages. Expect a delay of up to six weeks until your pages show up, assuming they are indeed accepted. This is not guaranteed to those using the Add URL page.

AllTheWeb will guarantee to include your pages if you use its paid inclusion program. It will get your new pages listed within 2 days. AllTheWeb doesn't sell paid inclusion directly to the public. Instead, inclusion is sold by AllTheWeb resellers. Currently, the best way to sign up is via the Lycos Insite Select program.

 

Getting Listed in Yahoo
Launched in 1994, Yahoo is the web's oldest "directory," a place where human editors organize web sites into categories. However, in October 2002, Yahoo made a giant shift to using Google's crawler-based listings for its main results.  If Yahoo is now powered by Google, then why bother using it?  For one thing, you might find that the way Yahoo "enhances" Google's listings with information from its own directory may make search results more readable. In addition, Yahoo's search results pages still show "Categories" links. When offered, these will take you to a list of web sites that have been reviewed and approved by a human editor.

It's also possible to do a pure search of just the human-compiled Yahoo Directory, which is how the old or "classic" Yahoo used to work. To do this, search from the Yahoo Directory home page, as opposed to the regular Yahoo.com home page. Then you'll get both directory category links ("Related Directory Categories") and "Directory Results," which are the top website matches drawn from all categories of the Yahoo Directory.

Yahoo has two submission options: "Standard," which is free, and "Yahoo Express," which involves a submission fee.  Why might you choose to pay when the free search engine submission option is available? Simply for a fast turnaround time. If you use the free submit choice, there's no guarantee that your submission will be reviewed quickly or at all.

As for commercial categories, Yahoo requires that sites pay a Yahoo Express submission fee of $299 per year. This fee doesn't guarantee that you will be listed, only that you'll get a yes or no answer about being accepted within seven business days. However, the vast majority of most decent sites are accepted.  If accepted, you'll be reevaluated after a year and charged the submission fee again, if you want to stay in Yahoo's commercial area. You should review the traffic you received from Yahoo over the past year, to decide if it is worth paying the fee again. If not, you can decline to be listed, and you will not be charged.

But what about Google? If you originally signed up with Yahoo hoping to influence Google, won't dropping your Yahoo listing cause you to be dropped by Google? Not necessarily. Google will keep listing your site on its own. Whether it will rank it well is a separate question. However, after a year of existence, your web site may have other important links pointing at it. This means that losing your link from Yahoo may not have much of an impact on your ranking. If money is tight, you could try dropping the Yahoo listing and only submit again if you find it does have an impact with how you rank in Google's results.

How do you submit? If you are submitting for free to a non-commercial category, click on the "Suggest a Site" link that appears at the bottom of category page. That will bring up a submission form. Fill it out, and you're done.

If you are paying to submit, you do not need to pick a category. Instead, just go to the Yahoo Express Submission Form:  : http://add.yahoo.com/fast/add?+Business

From there, Yahoo editors will choose a category for you. All you need to do is fill out the form that's presented.

Finally, even if you choose not to submit to the Yahoo Directory, you may still appear in some the results at Yahoo that come from Google.

 

Lycos
Lycos is one of the oldest search engines on the web, launched in 1994. It ceased crawling the web for its own listings in April 1999 and instead uses crawler-based results provided by AllTheWeb (see on this page). So why bother with Lycos rather than using the AllTheWeb.com site? You might like some of the features that Lycos provides.  "Fast Forward" lets you see search results in one side of your screen and the actual pages listed in another. Relevant categories of human-compiled information from the Open Directory appear at the bottom of the search results page. At the top of the page, Lycos will suggest other searches related to your original topic right under the search box. Perhaps you might even like the look and feel better! Whatever the reason, under the hood, Lycos provides all the same relevancy and comprehensiveness you'll find at AllTheWeb.com. 

Getting Listed: For the main editorial listings at Lycos, you need to be listed with AllTheWeb.com, which is described on this page. Paid listings come from Overture (see below), and additional paid listings come from Terra Lycos's own program.

 

Search Providers
The companies below are really in the business of providing search results to other people, rather than hoping you'll visit their own sites to search. They are listed here primarily to provide further explanation of how they partner with some of the search engines listed on this page.

Submitting To Inktomi
Among the major search engines, Inktomi is the second-oldest crawler.  However, the creators then formed their own company in 1996 with the same name and gained their first customer, HotBot, in the middle of that year. The company then pursued a strategy of "powering" other search engines, rather than running its own branded service for the public.  Today, Inktomi continues to crawl the web. It was purchased by Yahoo in March 2003.  Nevertheless, Yahoo-owned Inktomi still continues to provide results to Yahoo-rival MSN Search.  Getting listed: Read http://www.inktomi.com/products/web_search/submit.html

Submitting To LookSmart
LookSmart is a human-compiled directory of web sites. The company does operate its own web site, but this really isn't intended for the public to use. Instead, similar to Inktomi, LookSmart provides its results to other search engines that need listings.  LookSmart gathers its listings in two ways. Commercial sites must pay to be listed in its commercial categories, making the service very much like an electronic "Yellow Pages." However, volunteer editors at the LookSmart-owned Zeal directory also catalog sites into non-commercial categories for free. Though Zeal is a separate web site, its listings are integrated into LookSmart's results.  LookSmart launched independently in October 1996, was backed by Reader's Digest for about a year, and then company executives bought back control of the service.  Getting Listed: To submit for free, you must go to Zeal.com.  But, as with Yahoo, your submission to the non-commercial categories at Zeal must be for content that is truly non-commercial in nature. If you attempt to submit something that is actually commercial, it will be rejected.  How do you submit? You'll need to sign-up as a Zeal member.

Getting Listed in Overture
Formerly called GoTo until late 2001, Overture is an extremely popular paid placement search engine that provides ads to many of the search engines listed on this page. While Overture has traditionally been a paid listings provider, the company is expanding into offering crawler-based editorial results. To do this, it purchased AllTheWeb.  It allows sites to "bid" on the terms they wish to appear for. You agree to pay a certain amount each time someone clicks on your listing. This is why it is sometimes called a "pay-per-click search engine." For instance, let's say you wanted to appear in the top listings for "running shoes." You might agree to pay 25 cents per click. If no one agrees to pay more than this, then you would be in the number one spot. If someone else later decides to pay 26 cents, then you slip into the number two position. You could then bid 27 cents and move back on top, if you wanted to.

While some people go directly to the Overture web site to search, most people encounter Overture's paid listings via other search engines. For example, the very top listings for "running shoes" at Overture would also appear in the "Sponsored Links" section at the top of AOL Search's results. If your goal is to build visibility on search engines quickly, then Overture is an essential option for you to explore. No other route can put you in the top results of many major search engines in such a short period of time.  It is well worth it for anyone to open an Overture account and experiment with how paid listings may help them. An account requires a $50 minimum deposit, and you must spend at least $20 per month. By carefully selecting targeted terms, you can stretch that money out for one or two months and get quality traffic. When your initial deposit has expired, you may find that the editorial listings generated by your submissions to directories and crawlers have kicked in. This may mean that you can eliminate your ad spend with Overture entirely. On the other hand, you may find that you want to continue spending and perhaps even increase your budget, to target terms where you don't receive good editorial placement. You can sign-up for Overture's paid listings via the Overture (GoTo) Sign-Up Form http://www.overture.com/d/about/advertisers/.

 

 

You May Also Consider Submitting to AOL Search and Ask Jeeves

AOL Search
AOL Search provides users with editorial listings that come Google's crawler-based index. Indeed, the same search on Google and AOL Search will come up with very similar matches. So, why would you use AOL Search? Primarily because you are an AOL user. The "internal" version of AOL Search provides links to content only available within the AOL online service. In this way, you can search AOL and the entire web at the same time. The "external" version lacks these links. Why wouldn't you use AOL Search? If you like Google, many of Google's features such as "cached" pages are not offered by AOL Search.  Getting Listed: AOL essentially duplicates the editorial and ad listings that are shown on Google, so you need to be listed with Google (see above).


Ask Jeeves
Ask Jeeves initially gained fame in 1998 and 1999 as being the "natural language" search engine that let you search by asking questions and responded with what seemed to be the right answer to everything.  In reality, technology wasn't what made Ask Jeeves perform so well. Behind the scenes, the company at one point had about 100 editors who monitored search logs. They then went out onto the web and located what seemed to be the best sites to match the most popular queries.  Today, Ask Jeeves instead depends on crawler-based technology to provide results to its users. These results come from the Teoma search engine that it owns,.  Ask Jeeves also owns the Direct Hit service, but results from Direct Hit are no longer offered to the public directly through the Direct Hit site. 

Getting Listed: For the main editorial listings at Ask Jeeves, you need to be listed with Teoma.  As with Inktomi, Teoma has no free Add URL page -- but also as with Inktomi, Teoma crawls the web, so if you have links pointing at your web site, you may get included naturally.  They will guarantee to include your pages if you use its Ask Jeeves Site Submit paid inclusion program. It will get your new pages listed in a week.

 

Other Choices
The sites below are "major" in the sense that they either still receive significant amounts of traffic or they've earned a reputation in the past that still causes some people to consider them to be important. For various reasons explained below, they are not among our top search choices. However, certainly feel free to try them. They could turn out to be top choices for you.

Submitting To Alta Vista
AltaVista is the oldest crawler-based search engine on the web. It opened in December 1995 and for several years was the "Google" of its day, in terms of providing relevant results and having a loyal group of users that loved the service.  Sadly, an attempt to turn AltaVista into a portal site in 1998 saw the company lose track of the importance of search. Over time, relevancy dropped, as did the freshness of AltaVista's listings and the crawler's coverage of the web.  Today, AltaVista is once again focused on search. Improvements have been made, but crawlers such as Google and AllTheWeb provide more comprehensive results. Because of this, AltaVista is probably a third-choice crawler, one to try if you haven't found what you are looking for at one of its competitors.  AltaVista does remain strong is in terms of some of the specialty searching it offers. It provides a good image search service, and you can look for video and audio clips, as well. It also has an outstanding news search service. 

Getting listed:  As with the other crawlers, building links is the best way to get listed for free. However, AltaVista does take submission via the AltaVista Add URL page.  AltaVista will guarantee to include your pages if you use its AltaVista Express Inclusion paid inclusion program.  It will get your new pages listed in about a week to two weeks.

 

Netscape Search
Owned by AOL Time Warner, Netscape Search uses Google for its main listings, just as does AOL's other major search site, AOL Search. So why use Netscape Search rather than Google? Unlike with AOL Search, there's no compelling reason to consider it. The main difference between Netscape Search and Google is that Netscape Search will list some of Netscape's own content at the top of its results. Netscape also has a completely different look and feel than Google. If you like either of these reasons, then try Netscape Search. Otherwise, you're probably better off just searching at Google.  Getting Listed: Netscape essentially duplicates the editorial and ad listings that are shown on Google, so you need to be listed with Google.