Robots.txt

The 🤖 robots.txt application lets you add or modify rules in your robots.txt file. You'll be autonomous to deploy your modifications in just a few clicks. Welcome budget crawl optimizations 😉

The video tutorial

How to use the robots.txt application

Using robots.txt is easy 😉

To access the robots.txt application, connect to our EdgeSEO solution solution, then click on the Robots.txt application's configure button
Click on the "Replace robots.txt" toogle.
Add content to the field that appears on the screen. Copy and paste the rules from your robots.txt file.
Click on the save button.
The robots.txt file is deployed on your site

Frequently asked questions

Am I alerted to syntax errors when saving my rules?

keyboard_arrow_down

At present, we do not validate the code you enter in your robots.txt file.
We advise you to use a specialized validator.

What happens if I turn off EdgeSEO optimizations?

keyboard_arrow_down

When you turn off EdgeSEO optimizations, we continue to display robots.txt. If you wish to disable it, you'll need to switch the "Replace robots.txt" toogle back to off.