STW Help Icon

Adding / Removing Your URL

Before submitting your web page to any search engine you should always make sure the web page is search engine friendly. You can use the Meta Tag Analyzer tool to make sure your web pages are search engine friendly before you submit them to search engines.

Adding Your URL

Submitting your web pages to Scrub The Web is 100% free. No URL submission is guaranteed to be included in our index.

How do I submit to Scrub The Web?

You can use our Submit URL page to submit to Scrub The Web. You can also use our free on-line Easy Submit program to submit your web pages to Scrub The Web and other search engines.

Why do you require an email address?

The email confirmation process helps us prevent invalid and spam submissions which consume valuable system resources. We do NOT sell or use your email address for any other purpose other than this confirmation email.

How long does it take to get in Scrub The Web?

We are continually crawling our existing index as well as crawling new confirmed submissions around the clock. Usually you will see your site indexed within 7 to 14 days or sooner.

I keep submitting, but I never find my site. Why not?

If you can not find your own pages in search results then it is unlikely others will either. Therefore we highly recommend you use the Web Page Analyzer or Meta Tag Analyzer and see if there are possible problems. That said, Scrub The Web is not obligated to index your submitted pages. Our crawler selects URLs to be included in the index automatically without human intervention. Do not email us asking Scrubby to index your site as these emails will be ignored.

How do I update my listing once it is in Scrub The Web?

Scrub The Web will refresh the index automatically over time. There is no need for you to do anything. If you update your web pages Scrubby will detect the change when Scrubby re-visits the web page.

How does Scrub The Web determine which sites come up first when I do a search?

Each record gets a grade based on the relevancy of the search terms it contains when a searcher searches. For more information on our search technology click here.

How do I get my page to come up higher when someone is doing a search?

The best way is to optimize your website which will improve your rankings in all search engines.

Removing Your URL

How can I remove a page from Scrubby's index?

If you removed the page from your web server, we will also remove it from our search index when our crawler Scrubby detects the page no longer exists.

If instead you wish to exclude a single document from our index, you can create a text file at the root of your server called robots.txt. Example, let's say you want to exclude the document named "page-1.html" from all search engines. All you would do is place the robots.txt file in your server root and include the following:

User-agent: *
Disallow: /page-1.html

The above will prevent all search engines from including the "page-1.html" in their search results. If you wish to only exclude Scrubby from indexing this page, place the robots.txt file in your server root and include the following:

User-agent: Scrubby
Disallow: /page-1.html

You can also remove entire directories using this same method. This is the standard protocol that most web crawlers observe for excluding a web server or directory from an index. More information on robots.txt is available at site or Scrubby Prime Plus users can go here to learn efficient ways to better communicate with search engines.

There is another standard for telling robots to not index a particular web page or follow links on it, which may be more helpful for some users, since it can be used on a page-by-page basis. This method involves using the meta Robots tag.

Remember, changing your server's robots.txt file or changing the meta Robots tag will not cause an immediate change in what results Scrubby returns. It is likely that it will take a while for any changes you make to propagate to Scrubby's next index of the web.

For tips on how to improve your position on all major search engines including Scrub The Web, see our SEO Learning Center for details.

How To Not Get Listed

  1. Typographical errors. Make sure you enter the web address correctly. Double-check the spelling of the web address before you submit. Enter it in your browser to make sure it works. Spaces are NOT allowed in a valid URL.
  2. The website was inaccessible. Scrubby could not access the website after several attempts. The web server may be down or the site may have disappeared since you submitted it. Some host providers do a restart of servers and if this happens during the time Scrubby tries contacting your website, you will not be indexed.
  3. Banning entire websites! websites that employ practices such as unsolicited email may be excluded from our index without notice.
  4. Continued submissions without confirmation. Those who continue to submit to our index, but fail to confirm will eventually cause the email address and/or IP address to be placed in the opt-out (banned) list automatically.

Important: Remember that Scrub The Web is under no obligation to accept websites you recommend for inclusion in the index. We reserve the right to remove or exclude pages from the index at any time, for any reason, or for no reason at all.