Better alternatives to “Discourage search engines” in WordPress

This post picks up where a recent post left off: Arguing that WordPress’s “Discourage search engines” feature is best left unused because of the dire SEO consequences if you forget to disable it once a site is live. Here, we explore specific alternatives to “Discourage search engines” that don’t carry its risks.

We explore four approaches, in order of preference:

1. A full alternative that permanently replaces “Discourage search engines”;
2. A pragmatic alternative that also hides your development site from unwanted direct traffic;
3. A “lazy” alternative that is good enough for most projects; and
4. A few ways to make “Discourage search engines” safer if you absolutely must use it.

Enjoy!

1. Manually set robots.txt rules in your development environment

Create a development environment with search engines discouraged: anything inside it won’t get indexed, and anything outside it will.

What makes “Discourage search engines” scary is that it follows you from your development environment to your live site. You can fix that by manually creating a development environment with search engines discouraged. Anything inside the development environment won’t get indexed—and anything outside it will, like your live site once you make the transfer.