In my work as a WordPress consultant, only once have I actually damaged a site I worked on. I left “Discourage search engines from indexing this site” checked on a live site, suppressing the site’s page rank over the weeks until I noticed the mistake.

A single check box to wipe out hundreds of search rankings: it’s a startling asymmetry, especially since there’s no quick way to undo the damage.

Reflecting back on the experience, I’m most struck by the convenience: a single check box to wipe out hundreds of search rankings. It’s a startling asymmetry—especially since there’s no quick way to undo the damage even once you notice your mistake. Reindexing the site won’t yield results for hours, and regaining the lost page rank takes weeks and months, like restoring an old-growth forest.

So this post will cover what “Discourage search engines from indexing this site” does, why never to use it, and how to quickly find out if your site or your client’s site has used it. The hope is that more people can permanently avoid the trap of “Discourage search engines,” and that more sites with SEO suppressed can be spotted and rehabilitated.

What “Discourage search engines from indexing this site” does

“Discourage search engines from indexing this site” is a checkbox option in the Settings > Reading area of the WordPress admin.

Really understanding what that phrase means requires a bit of technical knowledge, so here are a couple of less technical ways to describe what the box does:

1. “Destroy this site’s SEO until further notice.”
2. “Make this site not show up in Google search at all.”

In more detail: Checking this box tells search engines to completely avoid inspecting the site’s contents, meaning that the site will not show up in search results.

The reason this feature exists is to prevent a site that shouldn’t show up in search from being found by search engines. Maybe the most common example is a site that’s under development but not ready to show to the public yet, which was why I checked the box in the anecdote above.

Why you should never use it

When you take your site live, your settings will be intact, and unless you specifically remember to uncheck the box, the site will bleed page rank.

It is far, far too easy to forget that you have checked “Discourage search engines from indexing this site.” When you take your site live, all your settings will be intact, and unless you specifically remember to uncheck the box, the site will stay invisible to Google and immediately start to bleed page rank.

If your site launches are like mine, they come with hundreds of last-minute details to attend to, many of which (hosting transfers, database migrations, DNS propagation, 301 redirects) are complex and often nerve-wracking. There’s likely also a fair amount of time pressure.

In all this tumult, WordPress’s warning that you’re about to cripple your live site’s page rank is terrifyingly quiet. Have a look at right, and see if you can spot it. (Remember, the image at right is taken from a dashboard six times this size full of text and images.)

The bottom line is that you shouldn’t use “Discourage search engines” because it’s very easy to leave it on too long, and the consequences of that are really bad.

What to do instead

Simply allowing your test sites to accumulate page rank should be fine for most projects.

I’ve written another article that goes deep into much better alternatives to “Discourage search engines.” For now, though: simply leaving the box unchecked, and allowing your test sites to accumulate page rank should be fine for most projects.

They should see almost no traffic in a reasonably short development process (say a month or less), and if you promptly delete them—or, better yet, 301 redirect them to the live site—at the end, you’ll be home free. If you forget, you may eventually notice the test site sharing page rank with the live site, which a 301 redirect will quickly fix.

If that’s not satisfying to you, see more detailed advice in our article on the topic.

How to quickly tell if your site is using “Discourage search engines”

If you manage only one WordPress site, check the “At a Glance” area of the Dashboard to make sure the text “Search Engines Discouraged” does not appear. If not, you’re all set. However, if you manage a portfolio of sites—especially client sites—you’ll want to quickly check each site, and you shouldn’t have to go to the trouble of logging in to each one (or firing up Webmaster Tools or anything else). Here’s how.

“Discourage search engines” changes the robots.txt file at the root of a WordPress installation. The screenshots below show two robots.txt files: one with search engines allowed (left) and one with search engines discouraged (right). The site on the left doesn’t want to let Google crawl /wp-admin/ or /wp-includes/, which is standard practice, since those directories aren’t where your front end site lives. The site on the right doesn’t want to let Google crawl anything.

Go to sitename.com/robots.txt for every site you’re worried about.

So you’ll just want to go to sitename.com/robots.txt for every site you’re worried about, and see what the file looks like. It’s much quicker than any other method I’m aware of.

It’s 3 AM. Do you know if your site exists in search?