Generate robots exclusion file (robots.txt) if site URL is a pathname
Currently, if the site URL value is a pathname (e.g., /docs
), the robots exclusion file (robots.txt) is not generated. However, this file does not require an absolute URL. Therefore, the file should be generated if site URL has any allowable value (which is the hint that the site is being published to a webserver, where this file becomes relevant).