Are you familiar with the robots.txt file? If you're an SEO, the answer is a resounding yes! This file is probably the most powerful tool available to SEOs for controlling and guiding the many bots that crawl your site. Of course, the one we're most interested in is Google bot. Although Bing has recovered a little since the advent of ChatGPT, for most of you, Google is still the place to be. 

You know the significant impact these few lines of code can have on the way your site is crawled. But it's sometimes difficult to access them simply or to identify the people on the tech side who can quickly modify them. As a result, you're not using robots.txt to its full potential. 

In this article, we'll take a look at good robots.txt practices and the mistakes you shouldn't make. At the end of the article, we'll also show you, with a demo, how to modify this file in just a few clicks using our EdgeSEO solution.

‍

Robots.txt: the starting point for an optimized crawl budget

As you already know, the robots.txt file is a text file used by websites to communicate with search engine crawlers aka "Google Bot". It provides guidelines on which areas of the site robots are and are not allowed to crawl. Positioned at the root of the domain, the robots.txt file acts as a guide for the robots, helping them to understand which parts of the site are accessible and which should be avoided, thus optimizing the exploration of your site. 

In other words, robots.txt should enable you to optimize your "crawl budget" to maximize the number of pages crawled by bots that are important to your business. Remember, the first step to achieving a top position in Google search results is for the bot to discover your pages. Without exploration, there's no indexing, and without indexing, there's no positioning and therefore no SEO traffic.

Best practices

Let's take a look at some good robots.txt practices. 

• The robots.txt file must be placed at the root of your website. For example, for www.nike.com/, the robots.txt file must be accessible at https://www.nike.com/robots.txt. This is a convention: if you don't place it at the root, it won't be taken into account. In the same way, you must scrupulously respect the robots.txt syntax.
  
• A robots.txt file contains groups of rules. Each group begins with a "User-agent" line specifying which crawler the rules apply to, followed by "Disallow" or "Allow" lines indicating which paths crawlers can or cannot crawl.
  

• You can use wildcards such as "*" to represent any number of characters, or "$" to indicate the end of a URL.
  

• Please note that the rules in the robots.txt file are case-sensitive. For example, "Disallow: /product.html" applies to "https://nike.com/produit.html" but not to "nike.com/PRODUIT.html".
  
• You can use the "Sitemap" directive to indicate the location of your XML Sitemap file. This can help crawlers discover your site's content more quickly. 
• You can use rules to block the exploration of specific file types, such as images or PDF documents.

But above all... mistakes to avoid

• The robots.txt file is public and readable by Internet users. Never use robots.txt to block access to sensitive or private information. As the file is public, this could expose such information. You can also block access to Internet users via the .htaccess file, as Fnac does https://www.fnac.com/robots.txt
  
• Use overly broad rules, such as Disallow: /, that block the entire site. This can prevent search engines from indexing your site. Or put the preprod robots.txt into production with Disallow: / (we see this one regularly 😉 ).
  
• Block CSS or JavaScript files that are essential for page rendering. This can prevent search engines from understanding and indexing the content correctly.
  
• Use conflicting rules: for example, use a Disallow rule to block a URL, then an Allow rule to allow it into the same user agent group.

If you'd like to find out more about robots.txt, you can consult Google's guide to robots.txt. You can also test the validity of your file on this page.

‍

Why is robots.txt so important in SEO? 

In SEO, robots.txt is important for optimizing your crawl budget. If the Google bot spends an hour a day on your site, your objective is to have it discover the pages you want to position in the search results. There's no need for it to crawl irrelevant pages. 

Unfortunately, it's not uncommon to discover when analyzing your logs that Google may be looping over pages that should be blocked in your robots.txt. And don't forget that if you give Google's bot the right information, it will be efficient and your overall indexing performance will be optimized. This is especially true if you manage sites with several million pages.

It can also help you save bandwidth on your servers by blocking certain robots that shouldn't be crawling your pages.

‍

Take back control of your robots.txt!

Setting up rules for your robots.txt is not complicated. You can ask your agency for advice on the right recommendations for your context. However, accessing and modifying it can be more complicated. Indeed, how many SEOs struggle to (let's not mince words) apply their modifications, whether due to the limitations of the CMS or the difficulty of identifying the right contact, without spending hours on it. A modification that takes a few minutes can turn into days or even weeks! 

If this applies to you, there are now solutions that enable you to regain control over your robots.txt and, more generally, your SEO roadmap. EdgeSEO lets you modify your site code directly "at the Edge", bypassing the technical limitations of your CMS. We provide you with a user-friendly dashboard for easy deployment of your SEO recommendations. This gives you greater agility and autonomy (and makes it easy to test all your optimizations).

If you'd like to get a head start on your competitors and implement our EdgeSEO solution, request a demo!

Adding or modifying your robots.txt file has never been so easy. In just a few seconds, you can deploy your rules and easily test new strategies.

‍