Skip to content

Why You Should Never, Ever Check “Discourage Search Engines”

In my work as a WordPress consultant, only once have I actually damaged a site I worked on. I left “Discourage search engines from indexing this site” checked on a live site, suppressing the site’s page rank over the weeks until I noticed the mistake.

A single check box to wipe out hundreds of search rankings: it’s a startling asymmetry, especially since there’s no quick way to undo the damage.

Reflecting back on the experience, I’m most struck by the convenience: a single check box to wipe out hundreds of search rankings. It’s a startling asymmetry—especially since there’s no quick way to undo the damage even once you notice your mistake. Reindexing the site won’t yield results for hours, and regaining the lost page rank takes weeks and months, like restoring an old-growth forest.

So this post will cover what “Discourage search engines from indexing this site” does, why never to use it, and how to quickly find out if your site or your client’s site has used it. The hope is that more people can permanently avoid the trap of “Discourage search engines,” and that more sites with SEO suppressed can be spotted and rehabilitated.

What “Discourage search engines from indexing this site” does

discourage search engines from indexing this site“Discourage search engines from indexing this site” is a checkbox option in the Settings > Reading area of the WordPress admin.

Really understanding what that phrase means requires a bit of technical knowledge, so here are a couple of less technical ways to describe what the box does:

  1. “Destroy this site’s SEO until further notice.”
  2. “Make this site not show up in Google search at all.”

In more detail: Checking this box tells search engines to completely avoid inspecting the site’s contents, meaning that the site will not show up in search results.

The reason this feature exists is to prevent a site that shouldn’t show up in search from being found by search engines. Maybe the most common example is a site that’s under development but not ready to show to the public yet, which was why I checked the box in the anecdote above.

Why you should never use it

When you take your site live, your settings will be intact, and unless you specifically remember to uncheck the box, the site will bleed page rank.

It is far, far too easy to forget that you have checked “Discourage search engines from indexing this site.” When you take your site live, all your settings will be intact, and unless you specifically remember to uncheck the box, the site will stay invisible to Google and immediately start to bleed page rank.

If your site launches are like mine, they come with hundreds of last-minute details to attend to, many of which (hosting transfers, database migrations, DNS propagation, 301 redirects) are complex and often nerve-wracking. There’s likely also a fair amount of time pressure.

search engines discouragedIn all this tumult, WordPress’s warning that you’re about to cripple your live site’s page rank is terrifyingly quiet. Have a look at right, and see if you can spot it. (Remember, the image at right is taken from a dashboard six times this size full of text and images.)

The bottom line is that you shouldn’t use “Discourage search engines” because it’s very easy to leave it on too long, and the consequences of that are really bad.

What to do instead

Simply allowing your test sites to accumulate page rank should be fine for most projects.

I’ve written another article that goes deep into much better alternatives to “Discourage search engines.” For now, though: simply leaving the box unchecked, and allowing your test sites to accumulate page rank should be fine for most projects.

They should see almost no traffic in a reasonably short development process (say a month or less), and if you promptly delete them—or, better yet, 301 redirect them to the live site—at the end, you’ll be home free. If you forget, you may eventually notice the test site sharing page rank with the live site, which a 301 redirect will quickly fix.

If that’s not satisfying to you, see more detailed advice in our article on the topic.

How to quickly tell if your site is using “Discourage search engines”

If you manage only one WordPress site, check the “At a Glance” area of the Dashboard to make sure the text “Search Engines Discouraged” does not appear. If not, you’re all set. However, if you manage a portfolio of sites—especially client sites—you’ll want to quickly check each site, and you shouldn’t have to go to the trouble of logging in to each one (or firing up Webmaster Tools or anything else). Here’s how.

“Discourage search engines” changes the robots.txt file at the root of a WordPress installation. The screenshots below show two robots.txt files: one with search engines allowed (left) and one with search engines discouraged (right). The site on the left doesn’t want to let Google crawl /wp-admin/ or /wp-includes/, which is standard practice, since those directories aren’t where your front end site lives. The site on the right doesn’t want to let Google crawl anything.

robots txt discourage search engines

Go to sitename.com/robots.txt for every site you’re worried about.

So you’ll just want to go to sitename.com/robots.txt for every site you’re worried about, and see what the file looks like. It’s much quicker than any other method I’m aware of.

discourage search engines and you're entering a world of painNow that you know what “Discourage search engines” does, take a quick look through your sites and make sure they’re okay.

And guys, please don’t use “Discourage search engines from indexing this site,” on your own sites or sites you build for others. Human beings make mistakes, and checking that box opens you up to one of the easiest and costliest mistakes you can make with a WordPress site.

Thanks for reading, and please share to spread the word!

Yay! 🎉 You made it to the end of the article!
Fred Meyer

32 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments
Erik Haagensen
March 24, 2016 6:34 pm

I’m using GoDaddy Managed hosting, and it appears that when you push a site from Live to Staging, they automatically check that checkbox for you. But when you push the development site from Staging out to Live, they don’t uncheck the box. Incomprehensible.

Solomon Izuchukwu
August 22, 2016 10:22 am

pls how can i removed wordpress1 from my domain name

Ivan Švaljek
August 17, 2016 12:21 pm

It would be a nice and fairly easy to implement wp core feature to notify (message bar / email) when a wp install changes it’s IP address and the tick is on.

GeorgiRG
August 15, 2016 12:27 pm

Thank you 🙂 !

Ayush Singh
July 4, 2016 5:30 pm

hey. thanks for helping me out . this article was good for newbs like me .happy blogging

D Horvath
June 2, 2016 8:44 am

I can’t uncheck this feature! Help!!

MathekJnr
May 26, 2016 5:36 am

I made the mistake but I unchecked before transferring the website to the live domain. But looks like it is taking too long to re-index.

Melvin
May 9, 2016 5:29 pm

hi I made this mistake and I have lost a lot. now I am barely recovering for the mistakes I make. if anyone can help me or even just a suggestion on what to do, please help. this is my website- http://www.americanhealthbio.com thank you and don’t make this mistake once cause it will cost you a lot specially if your website is a business website.

Stephen Prince
April 6, 2016 10:16 am

I’ve just fallen foul to this check box and it’s the kind of mistake I’ll only make once. I launched a shiny new WordPress website on behalf of my client thinking all was okay and hadn’t realised the developer had checked the box. To be honest I don’t use WordPress that often and didn’t even know this check box existed. Embarrassed and humbled I’ve offered my client a PPC budget until the websites recovered it’s rankings. This devil button is now firmly on my launch check list.

Virat Special Steels
March 9, 2016 7:26 am

sir I have written blog for my site but it not indexing by any search engine, we check it it not click on discourage

Kiersten
January 25, 2016 10:36 pm

I accidentally made my site live with “Discourage search engines” checked. I unchecked it today, will this still take effect? Or did I royally screw up?

fredclaymeyer
January 26, 2016 1:40 pm
Reply to  Kiersten

Robots will index your site starting now. Whether there’s “damage” depends on how long your site was live with search engines disabled, and (especially) whether it had prior rankings that disabling search engines could have negatively impacted.

If it’s a new site on a new domain and you just had the box checked for a couple of weeks, all you lost was the couple of weeks. 🙂

Make sure you go into Google Webmaster Tools and “Fetch as Google” — that will manually index your site and save you any delays going forward.

Christopher Hawkwind
January 23, 2016 10:39 am

Every professional should have a go live checklist, with ensuring robots aren’t disallowed being an important one, along with lots of others that are equally important.

toni
January 19, 2016 3:18 pm

The check mark for mine is greyed out and I can’t uncheck it. what should I do?

Tom Dupuis
November 30, 2015 3:44 pm

My client’s blog had 484 posts that were hosted on a subdomain which had this option checked. Just unchecked it… been working on their site for months now and was wondering what was going on. Goes to show that those “SEO checklists” can come in handy even if you think things like robots.txt is obvious.

fredclaymeyer
December 1, 2015 3:01 pm
Reply to  Tom Dupuis

Whew 😐

Huntertrips
March 9, 2015 7:32 am

Interesting article, thank you!
I hope somebody can help me now: while developing the new website, we moved the old one to another adress (just in case). I guess the old one damages the ranks of the new one, as a big part of the content is the same. Is checking the box on the old website enough to make it invisible to search engines and stop its impact on the new page?
While editing the new website, we used the box for a while. Is there something to do now to repair it?
Thanks for help.

Alina
December 18, 2014 4:01 am

thanks for share mate, this is very very interesting, I sometimes use this feature when I have a development server AND a production server, where the development server is publicly accessible, too (at a subdomain). So, we try to prevent the site form being discovered by checking this box. But, if you do a database migration from DEV to PROD, you can forget to check that setting. I’ve done it once.

lars hansson
March 6, 2015 10:57 am
Reply to  Alina

can someone help i have a customer who wanted a site developed. I
offered her hosting but she said she already had hosting. now her
hosting provider gave me access to her Cpanel and i tried to upload
WordPress no luck.

I notified them and they told me as they do not have the full license

wordpress needs to uploaded via FTP and they would do it. It did not
cross my mind to check the section Discourage search engines.

anyhow I started to build the site and when finished googled the site and saw the following error:

http://www.oharawells.com.au/
A description for this result is not available because of this site’s robots.txt – learn more.
You’ve visited this page many times. Last visit: 5/03/15

i then uploaded a sitemap generator to wordpress and get the following:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

anyhow this is the error I am getting when i google: ohara wells

http://www.oharawells.com.au/
A description for this result is not available because of this site’s robots.txt – learn more.
You’ve visited this page many times. Last visit: 5/03/15

could someone take a look and see what the hell is going on?

my email is webcactus@gmail.com

lars hansson
March 11, 2015 7:28 am
Reply to  lars hansson

cheers mate all is good from what I can see its now showing up

oharawells.com.au with the right description.

If I require Help in the future I will email you for some consultancy cheers

Lars

M Aamir
July 19, 2014 4:54 pm

By mistake i checked “Discourage search engines from indexing this site”

letter on i uncheck it but in web master tool and also wordsprss Dissallow

Alex Williams
March 19, 2014 7:31 pm

The simple thing would be for WordPress to show you a warning on every page that search engine indexing is not permitted. When you log in (each time you log in) you should have to see that message with the option to a) dismiss it once or b) dismiss it once and for all.

I sometimes use this feature when I have a development server AND a production server, where the development server is publicly accessible, too (at a subdomain). So, we try to prevent the site form being discovered by checking this box. But, if you do a database migration from DEV to PROD, you can forget to check that setting. I’ve done it once.

Better alternatives to "Discourage search engines" in WordPress | WPShout.com
February 4, 2014 5:28 pm

[…] post picks up where a recent post left off: Arguing that WordPress’s “Discourage search engines” feature is best […]

Philip
January 26, 2014 11:41 pm

As with anything, this is just one of the items to mark as complete on a site launch.
We have about 3 pages of things to check when we launch, and my SEO team will check this within an hour of going live.
One thing to note – my SEO team will often provide me a custom robots.txt file for the site launch, and this needs to be made sure it’s implemented. It goes along the lines of having all of the verification files / meta tags completed when launching a new site (as these likely won’t translate over from the old site).
All of my sites have alerts in analytics, if I see a large drop in organic search traffic, I should know within 24 hours.

Tauseef Alam
January 23, 2014 7:55 am

When my site is in development stage. I prefer to check this box. i still remember an incident when i forgot to uncheck it after i made the site live and it took me weeks to figure out why my site wasn’t index.

Michael
January 22, 2014 3:27 am

Sometimes I also can forget to check this option but when I used Yoast SEO, this plugin remaindered me that option was checked. (if we don’t delete this message). The notification was appeared on my wp admin header.

Piet
January 22, 2014 2:06 am

It’s good to remind people that leaving the box checked can have a negative effect, but I would not suggest not to use it, but instead adding it to your pre-launch checklist.

On top of that a site that has never been live yet cannot “… immediately start to bleed page rank”, for the simple reason there is no page rank to begin with.

Cassi
January 21, 2014 9:44 pm

I always use the “discourage search engines” checkbox when a site is in development. But I also have a set checklist of items that I look over before each launch. Unchecking the box is on my list, along with adding google analytics, and a couple of other items.

Jasper
January 21, 2014 8:50 pm

So you’re human, you make mistakes. But to blame WordPress is somewhat hyperbolic, and it is bad advice. Better advice to check this flag in your instance of WordPress.
Suppose you dial a wrong number on your phone, would you advise to never use a phone anymore?

Jason Cooke
March 9, 2016 8:37 am
Reply to  Fred Meyer

Take a chill pill Jasp, it’s a good article.

Or start the conversation in our Facebook group for WordPress professionals. Find answers, share tips, and get help from other WordPress experts. Join now (it’s free)!

32
0
Would love your thoughts, please comment.x

Most Searched Articles

Best JavaScript Libraries and Frameworks: Try These 14 in 2024

In this post, we look at the best JavaScript libraries and frameworks to try out this year. Why? Well, with JavaScript being available in every web browser, this makes it the most accessible programming language of ...

20 Best Free WordPress Themes for 2024 (Responsive, Mobile-Ready, Beautiful)

If you're looking for only the best free WordPress themes in the market for this year, then you're in the right place. We have more than enough such themes for you right ...

12 Best WordPress Hosting Providers of 2024 Compared and Tested

Looking for the best WordPress hosting that you can actually afford? We did the testing for you. Here are 10+ best hosts on the market ...

Handpicked Articles

How to Make a WordPress Website: Ultimate Guide for All Users – Beginners, Intermediate, Advanced

Many people wonder how to make a WordPress website. They’ve heard about WordPress, its incredible popularity, excellent features and designs, and now they want to join the pack and build a WordPress website of their own. So, where does one get ...

How to Start an Ecommerce Business: Ultimate Guide for 2024

Is this going to be the year you learn how to start an eCommerce business from scratch? You’re certainly in the right place! This guide will give you a roadmap to getting from 0 to a fully functional eCommerce business. ...