This post picks up where a recent post left off: Arguing that WordPress’s “Discourage search engines” feature is best left unused because of the dire SEO consequences if you forget to disable it once a site is live. Here, we explore specific alternatives to “Discourage search engines” that don’t carry its risks.
We explore four approaches, in order of preference:
- A full alternative that permanently replaces “Discourage search engines”;
- A pragmatic alternative that also hides your development site from unwanted direct traffic;
- A “lazy” alternative that is good enough for most projects; and
- A few ways to make “Discourage search engines” safer if you absolutely must use it.
Enjoy!
1. Manually set robots.txt
rules in your development environment
Create a development environment with search engines discouraged: anything inside it won’t get indexed, and anything outside it will.
What makes “Discourage search engines” scary is that it follows you from your development environment to your live site. You can fix that by manually creating a development environment with search engines discouraged. Anything inside the development environment won’t get indexed—and anything outside it will, like your live site once you make the transfer.
Hello,
I’m currently having a WordPress.com Blog (Free Plan) &
desire to sell my Digital Stuff online without setting-up a website and
relying wholly on the Blog, at first (have been short of funds). Thus,
going through with this thought I have made changes to my Blog, under
this after making the payment, the customer is redirected to my blog
page haivng a link to download the digital content. So, I just want the
payment gateway to be able to redirect the traffic to the Download page
on my blog and not have it searchable via search engines, etc. &
neither be it listed under the WordPress.com Posts list/Pages, etc.
because if the download page is searchable by the Search Engines &
listed under my Blog’s post, I wouldn’t earn anything because then the
customer would directly download the content without making the payment.
Regards,
Joojo.
[…] migration and no 301 redirects)? Is your marketing too timid (or did your developer never turn off “discourage search engines” in […]
[…] in the situation where your beta site has a better page rank than your live site, then you can redirect them to your live site once you have finished the development […]
[…] migration and no 301 redirects)? Is your marketing too timid (or did your developer never turn off “discourage search engines” in […]
Awesome follow. A great way to minimize risk and pick up some good advice on developing in tandem with live sites.
Keep ’em coming.
Thanks Fred!
I created a development site with a subdomain using a separate WP install. But after reading your article — just out of concern that checking “discourage search engines” on the dev site was affecting the main site — I disabled that option and made a `robots.txt`. Do you know if this option can affect other WP installs on the same server?
Thanks for writing, Ben! Discouraging search engines at subdomain.site.com shouldn’t affect search engine behavior at site.com itself–there’s a new robots.txt file for each subdomain, and they’re listened to individually. (As a note, “Disable search engines” also works by changing robots.txt, so you’re changing the same file either way.)
You con confirm that everything’s the way you want by visiting both subdomain.site.com/robots.txt and site.com/robots.txt, and making sure they’re both set the way you want. This would presumably look like “Disallow: *” for the subdomain, and like no special rule for the domain itself.
Does that help?
Yes! Thank you!