Generate a robots.txt file
Antora should be capable of generating a robots.txt file. While this is something the UI can handle (using static files), it seems reasonable to offer this as a core feature as well.
By default, this feature is disabled. If the following key is set in the playbook file, an "all access" robots.txt file will be added to the root of the site:
site:
robots: true
Alternately, this key can be used to specify the contents of the robots.txt file:
site:
robots: |-
User-agent: *
Disallow: /secret-component/
The robots.txt file will be added to the site catalog, perhaps by the site mapper (since this is a site map concern).