Skip to content

Better alternatives to “Discourage search engines” in WordPress

This post picks up where a recent post left off: Arguing that WordPress’s “Discourage search engines” feature is best left unused because of the dire SEO consequences if you forget to disable it once a site is live. Here, we explore specific alternatives to “Discourage search engines” that don’t carry its risks.

We explore four approaches, in order of preference:

  1. A full alternative that permanently replaces “Discourage search engines”;
  2. A pragmatic alternative that also hides your development site from unwanted direct traffic;
  3. A “lazy” alternative that is good enough for most projects; and
  4. A few ways to make “Discourage search engines” safer if you absolutely must use it.

Enjoy!

1. Manually set robots.txt rules in your development environment

Create a development environment with search engines discouraged: anything inside it won’t get indexed, and anything outside it will.

What makes “Discourage search engines” scary is that it follows you from your development environment to your live site. You can fix that by manually creating a development environment with search engines discouraged. Anything inside the development environment won’t get indexed—and anything outside it will, like your live site once you make the transfer.

Yay! 🎉 You made it to the end of the article!
Fred Meyer
Share:

8 Comments
Most Voted
Newest Oldest
Inline Feedbacks
View all comments
Eruption Joojo
December 24, 2015 7:11 am

Hello,

I’m currently having a WordPress.com Blog (Free Plan) &
desire to sell my Digital Stuff online without setting-up a website and
relying wholly on the Blog, at first (have been short of funds). Thus,
going through with this thought I have made changes to my Blog, under
this after making the payment, the customer is redirected to my blog
page haivng a link to download the digital content. So, I just want the
payment gateway to be able to redirect the traffic to the Download page
on my blog and not have it searchable via search engines, etc. &
neither be it listed under the WordPress.com Posts list/Pages, etc.
because if the download page is searchable by the Search Engines &
listed under my Blog’s post, I wouldn’t earn anything because then the
customer would directly download the content without making the payment.

Regards,

Joojo.

Why the Web Development Industry is a Jungle | WP Business Tips
October 16, 2014 4:46 pm

[…] migration and no 301 redirects)? Is your marketing too timid (or did your developer never turn off “discourage search engines” in […]

Managing a development and live site | Syllogic WordPress Development
August 18, 2014 3:33 am

[…] in the situation where your beta site has a better page rank than your live site, then you can redirect them to your live site once you have finished the development […]

Web development: It's a jungle out there | Press Up
April 16, 2014 6:24 pm

[…] migration and no 301 redirects)? Is your marketing too timid (or did your developer never turn off “discourage search engines” in […]

Mario M
February 5, 2014 8:54 pm

Awesome follow. A great way to minimize risk and pick up some good advice on developing in tandem with live sites.

Keep ’em coming.

Ben
February 4, 2014 6:39 pm

Thanks Fred!

I created a development site with a subdomain using a separate WP install. But after reading your article — just out of concern that checking “discourage search engines” on the dev site was affecting the main site — I disabled that option and made a `robots.txt`. Do you know if this option can affect other WP installs on the same server?

Ben
March 26, 2014 9:14 pm
Reply to  Fred Meyer

Yes! Thank you!

Or start the conversation in our Facebook group for WordPress professionals. Find answers, share tips, and get help from other WordPress experts. Join now (it’s free)!