Added a skeleton of best practices that will be fleshed out with more details as I work through them for the observability team.
Also this file has gotten long, so I'm adding a TOC.
Part of production-engineering#26457 (closed)