With this plugin you can add robots.txt fie and inform search engines and other crawlers: which pages and resources they can or can't request.
After installing the plugin:
Go to "Confluence administration Add > robots.txt > robots.txt"
Click Edit and update the desired contents of robots.txt file
Now, you sould see the robots.txt in the browser