Skip to content

A new hope: Documentation revamp in Yaook

TL;DR

  • Restructure all docs
  • Ditch mdbook, Sphinx all the things
  • Use the Diátaxis framework for structuring
  • Put all docs on docs.yaook.cloud; no more yaook.gitlab.io/arcane-path/nobody/can/even/remember.

Summary

I would like us to revamp and restructure our documentation. The ultimate goal is to have a single entrypoint (https://docs.yaook.cloud) with a beautiful page where users and developers can find what they need. The documentation should be up-to-date (eventually versionable) and heavily interlinked, with the lowest effort possible.

Personas

Before we get into the remainder of the issue, let me quickly clarify who I'm talking about. I'd like to define two personas:

  • Developer: contributes code or issue reports to one or more Yaook projects
  • User: deploys and maintains Yaook clusters (k8s or OpenStack)

And I'd like to define one role which is explicitly out of scope: The end-user of a cluster deployed with Yaook (be it k8s or OpenStack). That is the job of upstream documentation (kubernetes / OpenStack).

Note: I realize that we do need End-user documentation for yaook/k8s (e.g. wireguard setup), but I'd like to defer that for now. The end-user documentation for yaook/k8s might best live on a separate domain, because it is likely very specific to the C&H Managed Kubernetes product and likely a subset of the User documentation.

Motivation

Every time we onboard someone into Yaook, be it as a User or a Developer (see above for definition of these terms), there are a lot of pitfalls, unclarities, questions etc.. That is inefficient and frustrating.

In my opinion, there are three reasons:

  • Some documentation is simply lacking (e.g. a good tutorial (not just a quick start) for yaook/operator and yaook/k8s)

  • Documentation is scattered (yaookctl, metal-controller, installation-guide vs. docs.yaook.cloud / yaook.gitlab.io/k8s)

  • Different types of documentation are intermixed (conceptual explanations and technical references in yaook/operator instance), preventing each of the mixed components from getting the space and layout it needs.

    For instance, the API reference in yaook/operator is crammed down to just two(!) navigation entries, despite it having many thousand words. The same sphinx tree contains concept and handbook-type documentation, which makes search really inefficient for all of it.

Proposal

We restructure our documentation according to the diátaxis Framework (which is not a software or tooling framework, but a conceptual framework). A quick summary:

Diátaxis separates documentation among two axes: for studying ("getting to know something") vs. for working ("getting a task done") and practical vs. theoretical knowledge, which results in four types of documentation:

  • Tutorials (studying + practical)
  • How-to guides (working + practical)
  • References (working + theoretical)
  • Explanations (studying + theoretical)

Currently, we have mostly decent API and configuration reference docs, some more-or-less scattered how-to guides, a few scattered explanations and approximately zero tutorials (I mean it, read the definition of a tutorial and its distinction toward how-to guides over on Diátaxis).

From this, the following proposal is derived:

Structure

The following distinct and disjunct types of documentation should become first-class citizens in the Yaook project:

  • Tutorials geared toward Users
  • How-to guides geared towards Users
  • How-to guides geared towards Developers
  • Technical references geared towards Users (configuration reference, command line utility references, some CRD references)
  • Technical references geared towards Developers (API references, some CRD references)
  • Explanations geared towards Users (e.g. how scheduling keys work and why we chose that approach)
  • Explanations geared towards Developers (e.g. the concept described in yaook.statemachine.cue)

Presentation

The end-goal of this proposal is that we have a new https://docs.yaook.cloud which looks somewhat like this:

A clean overview page with seven cards: Tutorials, User Reference, Operation Guide, Developer Reference, Developer Guide, Understanding Yaook, and Understanding the Yaook Code

Tools

We use Sphinx for everything, which means that we migrate away from mdbook.

The reason for this is that Sphinx is extremely good at hyperlinking between different sets of documentation using intersphinx. It allows us to easily cross-reference for instance the Operation (= User) Guide for Yaook Operator to the Reference Documentation of yaook/k8s to provide details about how Rook can be installed. Or link from the Operation Guide to the Scheduling Key Concept Explanation and the Scheduling Key Reference for further details. Or from a Tutorial to the pages which explain concepts which can only be brushed over.

You get the idea. And all of that in a way which does not break on changes, because the links point to internal identifiers instead of file names and text titles.

The readability of Sphinx is achieved by using the Furo theme for anything except technical references, for which we stick to Alabaster or ReadTheDocs, which provide a better rendering of technical references than Furo does (and wants).

To reduce the cognitive load of writing reStructuredText, we load the MyST plugin, which provides extended Markdown syntax to Sphinx, while allowing to drop down to rST for more complex things.

Implementation

The rough idea is to create a docs.yaook.cloud repository with a Sphinx project containing the landing page. It loads references from all other documentation sections via intersphinx and provides shim pages which lead to those. Because it only contains shims, we disable the search there to not provide a misleading search bar.

The repository includes all other documentation sources as submodules. The submodules get updated automatically via a tiggered pipeline or by the renovate bot. All documentation, including the landing page, is built in this repository (the CI can be parallelized) and a single artifact is generated from that which encompasses all documentation in a single archive which is then deployed to docs.yaook.cloud.

Versioning

For now, out of scope. Once we have a proper release procedure in place, the docs.yaook.cloud repository should be versioned to the major releases so that we can have stable old documentation.

Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this issue are to be interpreted in the spirit of RFC 2119, even though we're not technically doing protocol design.

Landing page

  • MUST use Sphinx

  • MUST be updated by a CI job

  • MUST use renovate-bot, CI triggers, or a nightly cronjob depending on what is more suitable

    • It may turn out that renovate or CI triggers turn out to cause too much noise, in which case a nightly update job may be more appropriate
    • Especially given that we generally have pre-rendered docs available in other repositories

Developer reference

  • Describe APIs, python modules, code repository directory structures, etc.
  • MUST use Sphinx for Python
  • SHOULD use Sphinx for non-Python projects if possible
  • SHOULD live in the repository of the project it describes
  • SHOULD NOT contain cross-project content
  • MAY link to references of other projects
  • MAY link to the user reference of the same project
  • SHOULD use the ReadTheDocs or Alabaster theme
  • SHOULD point their intersphinx mapping at the new https://docs.yaook.cloud (to allow for working references in MR builds)

User reference

  • Describe command line interfaces, CRD structures, configuration options
  • SHOULD use Sphinx (see above for rationale)
  • SHOULD live in the repository of the project it describes
  • MAY be omitted if the tool is self-describing (e.g. yaookctl --help), but preferably tools like sphinx-click are used to also have it available in the standard (sphinx) format
  • SHOULD point their intersphinx mapping at the new https://docs.yaook.cloud (to allow for working references in MR builds)

Tutorials

  • Primarily aimed at users, not developers.

    • It's tricky to figure out what would be a good tutorial, like, write an operator? tricky.

  • Each tutorial MUST be its own Sphinx doctree, possibly even repository

    • That way, there is no distraction along the road when reading a tutorial
    • SHOULD provide a backlink to the tutorial overview (maintained in the docs.yaook.cloud repository) in the navigation with a custom template or so

Other content

  • MUST use Sphinx
  • SHOULD use the Furo theme
  • Placement TBD -- either in the project it relates to or in a documentation repository if no 1:1 assignment possible

Content ideas

Tutorials

  • "Tutorial: Install OpenStack Keystone in Kubernetes using Yaook"
  • "Tutorial: Install Kubernetes in OpenStack using yaook/k8s"

Developer How-tos

  • "How to write documentation"
  • "How to set up the development environment"
  • "How to contribute code"

User How-tos

  • "How to install a management cluster" (from installation-guide)
  • "How to deploy a node using metal-controller"
  • "How to deploy yaook/k8s from scratch on OpenStack"
  • "How to upgrade to a newer OpenStack release"
  • "How to recover a broken MariaDB cluster" (from installation-guide)
  • "How to use yaookctl to examine databases"
  • Other stuff from the installation guide

Developer Explanations

  • "Structure of yaook.statemachine"
  • "Structure of an Operator"
  • "Handling of stateful loads as a developer"
  • "What is cue layering and how does it work"

User Explanations

  • "How Yaook handles stateful workloads"
  • "How does Yaook do configuration management"
Edited by Steve Starke
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information