In my work as a WordPress consultant, only once have I actually damaged a site I worked on. I left “Discourage search engines from indexing this site” checked on a live site, suppressing the site’s page rank over the weeks until I noticed the mistake.
A single check box to wipe out hundreds of search rankings: it’s a startling asymmetry, especially since there’s no quick way to undo the damage.
Reflecting back on the experience, I’m most struck by the convenience: a single check box to wipe out hundreds of search rankings. It’s a startling asymmetry—especially since there’s no quick way to undo the damage even once you notice your mistake. Reindexing the site won’t yield results for hours, and regaining the lost page rank takes weeks and months, like restoring an old-growth forest.
So this post will cover what “Discourage search engines from indexing this site” does, why never to use it, and how to quickly find out if your site or your client’s site has used it. The hope is that more people can permanently avoid the trap of “Discourage search engines,” and that more sites with SEO suppressed can be spotted and rehabilitated.
What “Discourage search engines from indexing this site” does
“Discourage search engines from indexing this site” is a checkbox option in the Settings > Reading area of the WordPress admin.
Really understanding what that phrase means requires a bit of technical knowledge, so here are a couple of less technical ways to describe what the box does:
- “Destroy this site’s SEO until further notice.”
- “Make this site not show up in Google search at all.”
In more detail: Checking this box tells search engines to completely avoid inspecting the site’s contents, meaning that the site will not show up in search results.
The reason this feature exists is to prevent a site that shouldn’t show up in search from being found by search engines. Maybe the most common example is a site that’s under development but not ready to show to the public yet, which was why I checked the box in the anecdote above.
Why you should never use it
When you take your site live, your settings will be intact, and unless you specifically remember to uncheck the box, the site will bleed page rank.
It is far, far too easy to forget that you have checked “Discourage search engines from indexing this site.” When you take your site live, all your settings will be intact, and unless you specifically remember to uncheck the box, the site will stay invisible to Google and immediately start to bleed page rank.
If your site launches are like mine, they come with hundreds of last-minute details to attend to, many of which (hosting transfers, database migrations, DNS propagation, 301 redirects) are complex and often nerve-wracking. There’s likely also a fair amount of time pressure.
In all this tumult, WordPress’s warning that you’re about to cripple your live site’s page rank is terrifyingly quiet. Have a look at right, and see if you can spot it. (Remember, the image at right is taken from a dashboard six times this size full of text and images.)
The bottom line is that you shouldn’t use “Discourage search engines” because it’s very easy to leave it on too long, and the consequences of that are really bad.
What to do instead
Simply allowing your test sites to accumulate page rank should be fine for most projects.
I’ve written another article that goes deep into much better alternatives to “Discourage search engines.” For now, though: simply leaving the box unchecked, and allowing your test sites to accumulate page rank should be fine for most projects.
They should see almost no traffic in a reasonably short development process (say a month or less), and if you promptly delete them—or, better yet, 301 redirect them to the live site—at the end, you’ll be home free. If you forget, you may eventually notice the test site sharing page rank with the live site, which a 301 redirect will quickly fix.
If that’s not satisfying to you, see more detailed advice in our article on the topic.
How to quickly tell if your site is using “Discourage search engines”
If you manage only one WordPress site, check the “At a Glance” area of the Dashboard to make sure the text “Search Engines Discouraged” does not appear. If not, you’re all set. However, if you manage a portfolio of sites—especially client sites—you’ll want to quickly check each site, and you shouldn’t have to go to the trouble of logging in to each one (or firing up Webmaster Tools or anything else). Here’s how.
“Discourage search engines” changes the robots.txt file at the root of a WordPress installation. The screenshots below show two robots.txt files: one with search engines allowed (left) and one with search engines discouraged (right). The site on the left doesn’t want to let Google crawl /wp-admin/ or /wp-includes/, which is standard practice, since those directories aren’t where your front end site lives. The site on the right doesn’t want to let Google crawl anything.
Go to
sitename.com/robots.txtfor every site you’re worried about.
So you’ll just want to go to sitename.com/robots.txt for every site you’re worried about, and see what the file looks like. It’s much quicker than any other method I’m aware of.
It’s 3 AM. Do you know if your site exists in search?
Now that you know what “Discourage search engines” does, take a quick look through your sites and make sure they’re okay.
And guys, please don’t use “Discourage search engines from indexing this site,” on your own sites or sites you build for others. Human beings make mistakes, and checking that box opens you up to one of the easiest and costliest mistakes you can make with a WordPress site.
Thanks for reading, and please share to spread the word!
I’m using GoDaddy Managed hosting, and it appears that when you push a site from Live to Staging, they automatically check that checkbox for you. But when you push the development site from Staging out to Live, they don’t uncheck the box. Incomprehensible.
pls how can i removed wordpress1 from my domain name
It would be a nice and fairly easy to implement wp core feature to notify (message bar / email) when a wp install changes it’s IP address and the tick is on.
Thank you 🙂 !
hey. thanks for helping me out . this article was good for newbs like me .happy blogging
I can’t uncheck this feature! Help!!
I made the mistake but I unchecked before transferring the website to the live domain. But looks like it is taking too long to re-index.
hi I made this mistake and I have lost a lot. now I am barely recovering for the mistakes I make. if anyone can help me or even just a suggestion on what to do, please help. this is my website- http://www.americanhealthbio.com thank you and don’t make this mistake once cause it will cost you a lot specially if your website is a business website.
I’ve just fallen foul to this check box and it’s the kind of mistake I’ll only make once. I launched a shiny new WordPress website on behalf of my client thinking all was okay and hadn’t realised the developer had checked the box. To be honest I don’t use WordPress that often and didn’t even know this check box existed. Embarrassed and humbled I’ve offered my client a PPC budget until the websites recovered it’s rankings. This devil button is now firmly on my launch check list.
sir I have written blog for my site but it not indexing by any search engine, we check it it not click on discourage
I accidentally made my site live with “Discourage search engines” checked. I unchecked it today, will this still take effect? Or did I royally screw up?
Robots will index your site starting now. Whether there’s “damage” depends on how long your site was live with search engines disabled, and (especially) whether it had prior rankings that disabling search engines could have negatively impacted.
If it’s a new site on a new domain and you just had the box checked for a couple of weeks, all you lost was the couple of weeks. 🙂
Make sure you go into Google Webmaster Tools and “Fetch as Google” — that will manually index your site and save you any delays going forward.
Every professional should have a go live checklist, with ensuring robots aren’t disallowed being an important one, along with lots of others that are equally important.
The check mark for mine is greyed out and I can’t uncheck it. what should I do?
My client’s blog had 484 posts that were hosted on a subdomain which had this option checked. Just unchecked it… been working on their site for months now and was wondering what was going on. Goes to show that those “SEO checklists” can come in handy even if you think things like robots.txt is obvious.
Whew 😐
Interesting article, thank you!
I hope somebody can help me now: while developing the new website, we moved the old one to another adress (just in case). I guess the old one damages the ranks of the new one, as a big part of the content is the same. Is checking the box on the old website enough to make it invisible to search engines and stop its impact on the new page?
While editing the new website, we used the box for a while. Is there something to do now to repair it?
Thanks for help.
thanks for share mate, this is very very interesting, I sometimes use this feature when I have a development server AND a production server, where the development server is publicly accessible, too (at a subdomain). So, we try to prevent the site form being discovered by checking this box. But, if you do a database migration from DEV to PROD, you can forget to check that setting. I’ve done it once.
can someone help i have a customer who wanted a site developed. I
offered her hosting but she said she already had hosting. now her
hosting provider gave me access to her Cpanel and i tried to upload
WordPress no luck.
I notified them and they told me as they do not have the full license
wordpress needs to uploaded via FTP and they would do it. It did not
cross my mind to check the section Discourage search engines.
anyhow I started to build the site and when finished googled the site and saw the following error:
http://www.oharawells.com.au/
A description for this result is not available because of this site’s robots.txt – learn more.
You’ve visited this page many times. Last visit: 5/03/15
i then uploaded a sitemap generator to wordpress and get the following:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
anyhow this is the error I am getting when i google: ohara wells
http://www.oharawells.com.au/
A description for this result is not available because of this site’s robots.txt – learn more.
You’ve visited this page many times. Last visit: 5/03/15
could someone take a look and see what the hell is going on?
my email is webcactus@gmail.com
cheers mate all is good from what I can see its now showing up
oharawells.com.au with the right description.
If I require Help in the future I will email you for some consultancy cheers
Lars
By mistake i checked “Discourage search engines from indexing this site”
letter on i uncheck it but in web master tool and also wordsprss Dissallow
The simple thing would be for WordPress to show you a warning on every page that search engine indexing is not permitted. When you log in (each time you log in) you should have to see that message with the option to a) dismiss it once or b) dismiss it once and for all.
I sometimes use this feature when I have a development server AND a production server, where the development server is publicly accessible, too (at a subdomain). So, we try to prevent the site form being discovered by checking this box. But, if you do a database migration from DEV to PROD, you can forget to check that setting. I’ve done it once.
[…] post picks up where a recent post left off: Arguing that WordPress’s “Discourage search engines” feature is best […]
Hi Rhys,
Yes, I saw your plugin! I love it too–really a great idea.
I’m planning to mention it in a follow-up post on alternatives to “Discourage serach engines” (and best practices if you do decide to use it). But in the meantime, yes, your plugin is an awesome idea and everyone who uses “Discourage search engines” should get it!
When my site is in development stage. I prefer to check this box. i still remember an incident when i forgot to uncheck it after i made the site live and it took me weeks to figure out why my site wasn’t index.
Sometimes I also can forget to check this option but when I used Yoast SEO, this plugin remaindered me that option was checked. (if we don’t delete this message). The notification was appeared on my wp admin header.
It’s good to remind people that leaving the box checked can have a negative effect, but I would not suggest not to use it, but instead adding it to your pre-launch checklist.
On top of that a site that has never been live yet cannot “… immediately start to bleed page rank”, for the simple reason there is no page rank to begin with.
I always use the “discourage search engines” checkbox when a site is in development. But I also have a set checklist of items that I look over before each launch. Unchecking the box is on my list, along with adding google analytics, and a couple of other items.
So you’re human, you make mistakes. But to blame WordPress is somewhat hyperbolic, and it is bad advice. Better advice to check this flag in your instance of WordPress.
Suppose you dial a wrong number on your phone, would you advise to never use a phone anymore?
Thanks for writing, Jasper. I understand your point, but part of what I’m getting at is that WordPress is making it really easy to make a *very* big mistake.
To modify the analogy, what if the phone (let’s say it’s a cellphone) had a “blow up house” button—useful for blowing up your house behind you if you ever needed to make a quick escape, but scarily not-idiot-proof otherwise.
Obviously, the truth here is somewhere in the middle. But so far I believe (especially from my experience) that “Discourage search engines” is dangerous enough that using it isn’t worth the risk. I’ve started a follow-up post on good alternatives to it, the main one being manually modifying the robots.txt of your dev environment. Does the existence of safer alternatives sway your opinion at all?
Thanks again for writing!
Take a chill pill Jasp, it’s a good article.
As with anything, this is just one of the items to mark as complete on a site launch.
We have about 3 pages of things to check when we launch, and my SEO team will check this within an hour of going live.
One thing to note – my SEO team will often provide me a custom robots.txt file for the site launch, and this needs to be made sure it’s implemented. It goes along the lines of having all of the verification files / meta tags completed when launching a new site (as these likely won’t translate over from the old site).
All of my sites have alerts in analytics, if I see a large drop in organic search traffic, I should know within 24 hours.