Create separate Middleman instance for the Handbook (Part 2: remaining dirs)

What

Convert rest of handbook to be under sites/handbook and built by the existing dedicated handbook Middleman config and CI job.

How

Since we have now extracted the bulk of the middleman monorepo support logic to extensions/monorepo.rb, and it is driven by the data/monorepo.yml config file, most of the initial work consists of mostly rote process:

**UPDATE: Note there is now a /scripts/monorepo-migration-helper.rb to help with this process.

• Move the files.
• Keep running the sub-site middleman build and:
  • Moving missing files under sites/handbook/source/, if they are not used anywhere else, OR
  • adding entries to the data/monorepo.yml handbook site's shared_files section if they are used elsewhere...
  • ...until it passes.
• Run the top-level build, and ensure that no top-level pages reference pages which have been moved to the handbook (because we do not want circular dependencies). Hopefully, there are none. If there are, we need to (with the approval of the appropriate DRI) either:
  • Remove the reference from the top-level, or
  • Move the file down with the handbook.

Tasks

• Make necessary updates to sites/handbook/config.rb middleman config
• Move remaining proxies from top-level config to handbook config.
• Add/update development-mode-only files.watch references to handbook from top-level config.
• Update PartialBuild accordingly
• Understand why <li><a href="/handbook/">Handbook</a></li> started showing up, but doesn't show on master.
  • It's something about the logic in extensions/breadcrumbs.rb, which is obscure and not tested. The new behavior is better than the old (it should have had the extra link anyway, it's part of the hierarchy), so not worrying about it unless it causes problems.
• Understand why /handbook/business-ops/data-team/data-edw/index.html has extra HTML on master build.
  • It's because it has no frontmatter, so the branch version is not getting any enclosing HTML from the layout.
  • However, this indicates that the default layout isn't getting picked up properly for the branch build. Need to understand why.
  • Fix was to reintroduce explicit declaration of 'layout' as the default layout in the monorepo.rb extension. For some reason this has to be made explicit in the sub-site build, even though it's the default for Middleman. Could probably find the problem in middleman if we looked, but for now it's a simple one-line fix that will handle it for all sites.
• Understand why /handbook/customer-success/professional-services-engineering/workflows/workflow-list.html was not rendered on the branch.
  • It showed up after a rebuild. Only change was to fix the default layout, possibly related. 🤷
• Abstract out duplication across config files.
  • Possibly split into multiple common include files, to allow easier overriding? Only if it makes sense and doesn't seem like premature optimization.
  • Resolution: The proxy resources are now DRYd up. Going to wait until we extract a second site to DRY up the common config with the top-level.
• Remove any unnecessarily broad or non-targeted regex matches in shared_files config, and replace with individual filenames. For example, directories with multiple images are OK, and that's mostly what's left.
• Ensure CODEOWNERS was properly updated.
• Ensure that edit and web IDE links work at bottom of pages for moved pages. (we now have an automated CI check for this, but still spot check them all at least once)
• Remove any commented config sections
• Ensure dev mode, including proxy resources, all still work.
• Final pass of config, remove all unused commented code.
• Update handbook/changelog.rss autogeneration.
  • Changed REL_FILE_PATH = "sites/handbook/source/handbook/changelog.rss" in /bin/generate_handbook_changelog_rss
  • TODO: Need to verify it still actually works after merge.
  • Here's the changelog job run after the merge to master: https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/616741401
  • Update: It failed. The scripts for the changelog generation are duplicated (/bin/generate_handbook_changelog, which is duplicated from /bin/generate_handbook_changelog_rss), they have identical declarations of the path, and I only found and changed one of them. This is why we DRY. Fixed in !54577 (merged)
• Fix highlight-tooltip for files which live in a sub-site.
  • follow-on issue: #8128 (closed)
• Add some minimal docs on monorepo approach.
  • follow-on issue: #8129 (closed)
• Follow up on slack in #handbook after move: https://gitlab.slack.com/archives/C81PT2ALD/p1592984311343600

Additional unrelated tasks

Maybe do these as separate MRs, tracking them separately here for now:

• Fix all occurrences of File.expand_path("#{monorepo_root}/source", __dir__) (no necessary to expand if monorepo_root is used, because it's already an absolute path).

Related issues

Closes #7912 (closed)

/label ~"group::static site editor" ~backstage