Update robots file with current paths

What does this MR do and why?

We haven't done a review of the paths in the robots.txt file since !31782 (merged) and a few MRs from 2020-10 such as !46218 (merged).

This MR mostly adds paths from the main group/project sidebar navigation, while removing a couple (such as deploy_keys) that no longer exists.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Resolves #478252 (closed)