- [x] Visiting a directory without a trailing / causes any relative links in that directory to fail - works locally, but not in prod because prod url has .domain, local does not. Look at Request.swift where it sets the forceRedirect flag - [x] Move `requestInterval` property in the config file out of the "static" section; that section is intended for configuring static *content* only - [x] Rename `static` property in config file to `staticContent` - [x] Redesign the `ini` example below in `json`; `ini` would require implementing a custom parser - [x] Enhance `config.json` with support for: - [x] Update README.md ```json "shouldServeRobotsFile": true ``` - Gemini's adaptation of the web's de facto standard robots.txt - For controlling access to content by bots - robots.txt states desired bot policy in machine readable format - Authors of bots are strongly encouraged to check for such policies and to comply with them when found - Server admins should understand that it is impossible to enforce a robots.txt policy and must be prepared to use firewall rules, etc. to block access by misbehaving bots - This is equally true of Gemini and the web - [x] Lives at root / - gemini://example.net/robots.txt - [x] Should be served with a MIME media type of text/plain - Format is as per the original web spec for robots.txt: - Lines beginning with # are comments - Lines beginning with "User-agent:" indicate a user agent to which subsequent lines apply - Lines beginning with "Disallow:" indicate URL path prefixes which bots should not request - All other lines are ignored - Only non-trivial difference between robots.txt on the web and Gemini is that, because Gemini admins cannot easily learn which bots are accessing their site and why (because Gemini clients do not send a user agent), Gemini bots are encouraged to obey directives for "virtual user agents" according to their purpose/function (described below) - Regardless, Gemini bots should still respect - robots.txt directives aimed at a User-agent of \* (less the \) - May also respect directives aimed at their own individual User-agent which they, e.g., prominently advertise at the Gemini page of any public services they provide - A common category of bot - Bots should respect directives aimed at any virtual user agent which matches their activity - If bot meets multiple virtual user agent definitions, it should obey the most restrictive set of directives arising from the combination of all applicable virtual user agents - Fetch content to build public long-term archives of Geminispace - Will serve old Gemini content even after the original has changed or disappeared (think "Wayback Machine" - Should respect robots.txt directives aimed at a User-agent of "archiver" - Fetch content to build searchable indices of Geminispace - Should respect robots.txt directives aimed at a User-agent of "indexer" - Fetch content to study large-scale statistical properties of Geminispace (page/domain counts, distribution of MIME media types, response sizes, TLS versions, frequency of broken links, etc.), without rehosting, linking to, or allowing search of any fetched content - Should respect robots.txt directives aimed at a User-agent of "researcher" - Fetch content to translate said content into HTML and publicly server the result over HTTP(s) - Should respect robots.txt directives aimed at a User-agent of "webproxy"