Update robots.txt to Allowlist instead of Denylist

Description

In our robots.txt we list out every route with Disallow: .... As seen in this MR, this can easily become outdated and require frequent updates.

Let's consider using an allowlist approach where we initially Disallow: /*/ and only Allow: ... the paths that we want to be readable by robots.

Context

The following discussion from !162349 (merged) should be addressed:

• @pslaughter started a discussion: (+2 comments)

  question: Do we want to try doing an allowlist approach instead of this denylist approach? Would something like this work:

  Disallow: /*/-/
  Allow: /*/-/issues/
  Allow: /*/-/merge_requests/
  Allow: /*/-/epics/

  WDYT? Maybe this is a follow-up 🤔