Overview #
WordPress.com uses the core robots.txt
file and adds a number of default entries, such as the sitemap, to optimize your site. To modify the file, you can hook into the do_robotstxt
action, or filter the output by hooking into robots_txt
(source).
Example: Mark a directory as “nofollow” #
function my_disallow_directory() { echo "User-agent: *" . PHP_EOL; echo "Disallow: /path/to/your/directory/" . PHP_EOL; } add_action( 'do_robotstxt', 'my_disallow_directory' );
Caching #
Note that we cache the robots.txt for long periods of time. This means that you’ll need to force the caches to clear after any changes, by going to Settings > Reading
from your Dashboard and toggling the privacy settings.
On convenience domains #
On any convenience domain (a subdomain of go-vip.net
or go-vip.co
), the robots.txt output will be hard-coded to return a “Disallow for all user agents” result. This is to prevent search engines from indexing content hosted on development/staging sites.