Scrubby runs on a combination of advanced hardware and software. The experience you get when visiting Scrub The Web can be attributed in part to the efficiency of our programming and partly to the Linux based servers we've networked together to create a fairly fast Website.
Scrub The Web was created in part by open source programs such as Apache and Linux. While these are the foundation of our server technology, modifications to these core elements were necessary. The heart of our system was developed by Anthony Federico who did most all the coding on the Linux kernel and Apache source. He is also responsible for the indexing and search technologies. And while we have other engineers working to improve Scrubby, the work Anthony has done and continues to do is the basis for everything you see here.
Scrub The Web's search engine uses an inverted index to enhance search speed. Scrub The Web runs on multiple server nodes and each can index and search more than 1.5 billion documents very quickly. A single indexer can easily fetch and index more than 1.5 million documents in 24 hours. New technologies we are working on will allow us to index and search tens of billions of documents.
While search engines like Google concentrate on quantity and rank results by popularity and advertising revenues, Scrub The Web's focus is in helping new sites be found in not only our search engine, but in others as well. We are not trying to index the entire Web and waste your time by delivering millions of pages all with basically the same content. Our goal is to deliver a diverse group of results targeted to your search query and rank those results by content without any other ranking algorithms like popularity or how much advertising revenue a site generates.
We feel popularity is not the most important aspect of delivering relevant search results. While some search engines feel popular means relevant, we feel popular only means the not yet popular site "can't become popular" because no one can find them.
Instead of playing the popularity game, we decided to let all indexed pages stand on their own. If the search terms are found on the page it is relevant. The ranking of relevance is another story. Scrubby takes many aspects into consideration. The Title and Meta Description Tags are important and so is the content found on the page. In fact any viewable content in our opinion is considered relevant if it contains the search terms you are looking for. Popularity with little content will not usually rank well at Scrub The Web.
Popularity is based on the number of links from page A to page B. If page A is also popular, Page B will achieve a slightly higher ranking than those without links or links from less popular sites. The more popular the sites are linking to you, the higher rank you will gain. Here at Scrub The Web we do not play that game. Instead well structured HTML and relevant content is king.
One of the biggest flaws with a popularity ranking algorithm is that it's an easy algorithm to manipulate and exploit. With the price of a domain name being just a few dollars and hosting costs being very inexpensive as well, it's now easy and cheap to create artificial popularity. This single algorithm is what new site owners find difficult to overcome and feel they can't compete since they can't be found in the other search engines.
Scrub The Web does not sell placement within the results themselves (i.e., no one can buy a higher ranking). A Scrubby search is an easy, honest and objective way to find high-quality Websites with information relevant to your search.