STW Help Icon

Our Search Engine Technology

Scrubby runs on a combination of advanced hardware and software. The experience you get when visiting Scrub The Web can be attributed in part to the efficiency of our programming and partly to the Linux based servers we've networked together to create a fairly fast search experience.

Introduction

Scrub The Web's search engine uses an inverted index to enhance search speed. Scrub The Web runs on multiple server nodes and each can index and search more than 1.5 billion documents very quickly. In fact, a single indexer can easily fetch and index more than 1.5 million documents in less than 24 hours.

Scrub The Web's focus since the beginning has always been to help new websites be found in the search results of all search engines. In fact, the Scrub The Web search engine was originally created so we could share with our visitors how search engines work. At Scrub The Web we focus on delivering a diverse group of results that match your search query and then rank those results by their relevance to the query. We do not use silly ranking algorithms like a site's popularity when generating results. Instead we concentrate on the content found.

Scrubby's Ranking Explained

We feel popularity is not the most important aspect of delivering relevant search results. While some search engines might feel popular means relevant, we feel popular only means the not yet popular "can't become popular" because no one can find them.

Instead of playing the popularity game, we decided to let all indexed pages stand on their own without regards to popularity. Scrubby takes many other aspects into consideration when ranking results. The Title of the page and how the page is described using the meta Description are important and so is the content found on the page. Popularity with little content will not usually rank well at Scrub The Web.

Popularity Briefly Explained

Popularity is calculated using the number of links from one website to another. For example, if Site A places a link on their home page to the home page of Site B, it means Site A has made the home page of Site B more popular. The more links Site B gets from other sites on the Internet, the more popular Site B will become. It is this nasty ranking algorithm which causes website owners to receive dozens of emails in their in-box daily from people wanting to trade, sell and buy links.

One of the biggest flaws with a popularity ranking algorithm is that it's an easy algorithm to manipulate and exploit. With the price of a domain name being just a few dollars and hosting costs being very inexpensive as well, it's now easy and cheap to create artificial popularity by linking inexpensive sites all owned by the same person. This single algorithm is what new site owners find difficult to overcome and feel they can't compete since they can't be found in the other search engines.

Scrub The Web does not sell placement within the results themselves (i.e., no one can buy a higher ranking). A Scrubby search is an easy, honest and objective way to find high-quality search results relevant to your search.