Keeping the Search Engines from Crawling Your Website

By: | Updated: October 14, 2015

Our website is supported by our users. We sometimes earn affiliate links when you click through the affiliate links on our website.

Sometimes, when you’re not ready for the search engines to find your website, you may want to keep the Search Engines from Crawling Your Website. This may be the case for good with some website, but maybe just temporarily with others. There are ways to keep the search engines from crawling your WordPress website. Here is how you can stop them.

Why Should You Stop the Search Engines from Seeing Your Website

It may seem counterproductive to keep the search engines from crawling your website, especially if you want more visitors. However, there are times when it makes sense to keep them away.
If you plan to use the website as a development website, you may want to keep the search engines from crawling. Another reason to do this is if you plan to keep your blog private or it’s a site dedicated to just you and your family. You may not want to give access to strangers in this case. This is the easiest way from Keeping the Search Engines from Crawling Your Website.
Don’t think that just because there are not any links pointing to your website, it cannot be found by search engines. Your site could be found because the domain was previously owned by somebody else or because your site shows up in some directory you don’t even know about. Search engines know how to find websites, so you need to block them, if you don’t want them to find yours.

Search Engines from crawling

Blocking the search engines in WordPress is quite simple. This CMS comes with a built-in feature specifically for this task. Go to Settings>>Reading and you will see a section called, “Search Engine Visibility.”

Check the box and click the “Save Changes” button. This will add code to your website’s header that will make your site a “noindex,follow” website. This code tells the web crawlers or robots not to index your pages or posts.
Most search engines will honor this request, but some will ignore it. You may find that some random pages or images will get indexed, but this will take care of most of the search engines.

How to Make Sure You Don’t Show Up in Any Search Results

If you want to make sure none of your pages, posts or images show up on any search engine, you need a different level of security. The best way to go about blocking all the search engines is by adding password protection to your entire WordPress website. This is done from the cPanel of your hosting account on the server level.
When you add password protection, the search engine crawlers will see a 401 error page after login failure and will turn away.

How to Password Protect your WordPress Website

You will need to login to your cPanel to do this. Once you’re in your cPanel, you will need to find the “Security” section and look for the “Password Protect Directories” option.

Click on this option and you will get a popup menu.

From this menu, you need to choose the document root for your website.
After this is done, you will need to select he folder you want to protect for your WordPress website. If you want to block the entire site with password protection, select the public_html or www directory for your website.

Now, you will need to check the “Password protect this directory” option and add a name for the protected directory.

After you have done this, you can add a new username and password to access the website.

Once you’ve done this and saved it, you will see a message saying it was a success. Then, if you go to your website URL, you will have to log in to actually access the website.
Whether you want full protection from the search engines indexing your WordPress website or you just want to keep it blocked until it’s ready to launch, you can use these two options to keep the search engines from indexing your website.


Leave a Comment