Submitting your web pages to Scrub The Web is 100% free. No URL submission is guaranteed to be included in our index.
The email confirmation process helps us prevent invalid and spam submissions which consume valuable system resources. We do NOT sell or use your email address for any other purpose other than this confirmation email.
We are continually crawling our existing index as well as crawling new confirmed submissions around the clock. Usually you will see your site indexed within 7 to 14 days or sooner.
If you can not find your own pages in search results then it is unlikely others will either. Therefore we highly recommend you use the Web Page Analyzer or Meta Tag Analyzer and see if there are possible problems. That said, Scrub The Web is not obligated to index your submitted pages. Our crawler selects URLs to be included in the index automatically without human intervention. Do not email us asking Scrubby to index your site as these emails will be ignored.
Scrub The Web will refresh the index automatically over time. There is no need for you to do anything. If you update your web pages Scrubby will detect the change when Scrubby re-visits the web page.
Each record gets a grade based on the relevancy of the search terms it contains when a searcher searches. For more information on our search technology click here.
The best way is to optimize your website which will improve your rankings in all search engines.
If you removed the page from your web server, we will also remove it from our search index when our crawler Scrubby detects the page no longer exists.
If instead you wish to exclude a single document from our index, you can create a text file at the root of your server called robots.txt. Example, let's say you want to exclude the document named "page-1.html" from all search engines. All you would do is place the robots.txt file in your server root and include the following:
The above will prevent all search engines from including the "page-1.html" in their search results. If you wish to only exclude Scrubby from indexing this page, place the robots.txt file in your server root and include the following:
You can also remove entire directories using this same method. This is the standard protocol that most web crawlers observe for excluding a web server or directory from an index. More information on robots.txt is available at robotstxt.org site or Scrubby Prime Plus users can go here to learn efficient ways to better communicate with search engines.
There is another standard for telling robots to not index a particular web page or follow links on it, which may be more helpful for some users, since it can be used on a page-by-page basis. This method involves using the meta Robots tag.
Remember, changing your server's robots.txt file or changing the meta Robots tag will not cause an immediate change in what results Scrubby returns. It is likely that it will take a while for any changes you make to propagate to Scrubby's next index of the web.
For tips on how to improve your position on all major search engines including Scrub The Web, see our SEO Learning Center for details.
Important: Remember that Scrub The Web is under no obligation to accept websites you recommend for inclusion in the index. We reserve the right to remove or exclude pages from the index at any time, for any reason, or for no reason at all.