Google is really good at finding, crawling, and indexing websites – that’s their core business. The trick is, sometimes you don’t want them to.
If you’re redesigning your site, your designer may set up a “development” or “dev” site at a temporary location. That’ll let you see their design and make changes before it goes “live.” The domain might look something like dev.blogtutor.com, or if it’s on their own domain, perhaps blogtutor.webdesigncompany.com.
Or, you might maintain a clone of your site for ongoing testing — to check plugins for compatibility before upgrading, for example.
It’s really important that you keep Google (and other search engines) from indexing development, testing, or staging sites – otherwise, you may end up with pages from that domain in the search results. That can cause duplicate content issues, since you’ll probably have many posts & pages that were copied over from your real site.
In a worst-case scenario, if Google decides that the dev site is the authoritative version of your site, it will start showing it in the search results instead of your main site.
This is a disaster. Traffic will start going to the dev site, and then once you take down the dev site, all those links will be broken, leaving you with nothing — and your main site might never recover on its own. I ran into this on a client’s site eight months ago, and Google still has a handful of the development site URLs in the index, even with the correct fix in place.
Block Search Engines on a Test Site
There are a few ways you can prevent this from happening. The easiest and fastest is to use a built-in WordPress setting to block search engines. In the test site’s dashboard, go toand check the box next to “Search Engine Visibility.”
This tells WordPress to add a “noindex” tag in your website’s <head> section, which in turn tells Google not to index the content. It also modifies your site’s robots.txt file to tell bots they’re not welcome here.
But this comes with one huge caveat: If you ever turn the secondary site into your main site (as is often the case with a redesign), you must remember to un-check that box!! If you don’t, you could end up accidentally killing all of your real site’s positioning in Google.
Because that is such an easy mistake to make, and comes with extreme consequences, I do not recommend using the above technique for a development site that will be copied over to become the live site. It’s just too easy to screw that up.
Block Search Engines on a Development Site
So, for a development site that may be copied over at some point in the future, a better solution is to password protect the entire site. Not only will this keep Google from ever seeing the site – it’ll also keep real people (other than those you want, of course) from seeing it, too.
The ideal way to password protect a development site is at the server level. Each web hosting company has a slightly different way of setting this up, but if your host uses cPanel, this short video will walk you through it. (Or you could just open a support ticket and ask your host to do it for you.)
Alternatively, I recommend the Password Protected plugin as a quick and easy method to password protect your development site. It has a settings page to allow you to set a password, and once you enable it, you’ll have a nice-looking password entry form, like so:
Of course, once you copy over the development site, you’ll need to deactivate and remove that plugin — but it’s a lot easier to remember to do that.
And if you didn’t do all of the above, here’s how to fix it if Google already indexed your development site.