Most website owners want search engines like Google to index their website. However, we also receive questions from WordPress users that don’t want Google to “crawl” or index their website. In this article we’ll explain how to stop Google from indexing your website.
As said, many webmasters – me included – do want Google to index their website. You may wonder why anyone would want to block search engines from crawling their websites. However, there are lots of users that work on their websites on publicly accessible domains. Others use WordPress as a project management website or internal blogging platform. In these scenarios you probably do not want Google or other search engines to index your website.
Block Google (and other search engines) from crawling and indexing your WordPress website
For users that do not want their website to become visible in search engines like Google, Bing or Yahoo, there is a settings page in WordPress that allows them to block search engines. To reach this page, go to Settings >Reading. You will find an option here called “Search Engine Visibility“. It will show a checkbox called “Discourage search engines from indexing this site”.
Once this checkbox is checked and you have saved your changes, WordPress will add a line of code to all your website’s pages:
<meta name='robots' content='noindex,follow' />
This line of code including the “noindex” tag tells search engines not to index your page. In addition, WordPress changes your robots.txt file to include the following:
The robots.txt file is used by search engines as a reference. By including the above lines in your robots file, you are telling search engines not to index or crawl any pages on your website. Please note that this does not apply to any pages that are on a subdomain. It is only used for pages and URLs on your main domain.
It is up to search engines to honor this request
As you can read in the screenshot above, WordPress tells you that it is up to search engines whether or not to honor your request. While this is true because your website wil still be publicly accessible, in my experience most search engines definitely honor your request.
How long does it take for my website to disappear from Google?
After making the changes above, it may take weeks (!) for Google to stop showing your website in the search results. There is really no way to speed up this process. Therefore, if there is very sensitive information on your website, I would recommend to add password protection to your website.
Password protection for your WordPress website
To protect your entire WordPress website with a password, you can use a free plugin such as Password Protected. Though it protects all your posts and pages with a password, it does not protect the files you may have uploaded to your WordPress media library.