Welcome to another Teaching Tuesday!
Every website should have a robots.txt file regardless of whether it’s running on a content management system or not. In this post we’re going to deal specifically with WordPress websites because they are so popular and we’re very familiar with building and optimising them.
#1 Allow Robots
The very most important matter is ensuring that your website allows all robots (search engines, etc) to access your website so that your website may appear in the search results. To do this is really straight forward, it is just one line of writing:
User-agent: *
#2 Hiding /wp-admin/ and /wp-includes/
WordPress automatically creates three folders on the server, /wp-admin/, /wp-content/ and /wp-includes/. The /wp-admin/ and /wp-includes/ folders are system folders and we do not need Google and other search engines to index them, so instead of wasting the search engines’ precious time, let’s tell the robots not to include them, like so:
Disallow: /wp-admin/
Disallow: /wp-includes/
#3 Sitemap
Another file every website should have is a sitemap. With WordPress, most of the SEO plugins will automatically create the sitemap file. This sitemap should be included in your robots.txt file, this is another way to ensure that the robots and search engines are accessing all the important areas of your website and thus including the correct information in their indexes. The following code is used to achieve this:
Sitemap: http://www.yourdomain.com/sitemap.xml
Simply replace the link above with the link to your sitemap.
EXTRA
#1 Blocking a Search Engine
If for some reason you want to stop a specific robot or search engines from crawling your website, the following code may be used:
User-agent: Googlebot
#2 Blocking a Folder
If for some reason you want to stop a folder from being crawled, the following code works:
Disallow: /foldername/
You may repeat the line above multiple times if you want to block multiple folders as shown in #2 above.
ADDITIONAL RESOURCES
Once you have created your robots.txt file, simply upload it to the top level directory on your server. This is often the /public_html/ folder or the folder that contains your index.html / index.php file.
If you have any questions, perhaps about meta robots and the likes, please leave us a comment and we will answer it with pleasure.
