Tracking Issue: Implement GLFM scripts per the specification guide

Overview

This is the single Tracking Issue for the following tasks. Each individual task will only have MRs to represent the work, not separate issues. See the following thread for more discussion on how we arrived at this approach: &7719 (comment 933806489)

See the parent epic for other Tracking Issues: GitLab Flavored Markdown (GLFM) Specification D... (&7719)

See the GLFM Specification Guide development documentation for much more context on all of this work: https://docs.gitlab.com/ee/development/gitlab_flavored_markdown/specification_guide

Tasks

Main Implementation Tasks

• Implement GLFM update-specification.rb script
  • Issue: Implement GLFM `update-specification.rb` script (#360212 - closed)
  • MR: Implement GLFM `update-specification.rb` script (!85640 - merged)
• Implement GLFM update-snapshots.rb script (see old issue, closed in favor of this one):
  • MR: Implement GLFM `update-example-snapshots.rb` sc... (!84347 - merged))
• Add proper support for all new scripts and files to https://gitlab.com/gitlab-org/gitlab/blob/master/tooling/danger/project_helper.rb. See thread with details: !84347 (comment 936095519):
  • MR: Update tooling projects_helper to handle scripts (!87745 - merged)
• Add UPDATE_GHFM_SPEC_TXT env var, and only download latest GHFM spec.txt when it is set. Otherwise, use the current version committed to the repo. This will be a guard against potential injection attacks if the canonical GFM spec.txt hosted by GitHub is temporarily compromised. See related threads here and here:
  • MR: Only download GHFM spec.txt when specified (!87748 - merged)
• Finish support for using glfm_example_status.yml to prevent overwriting manually-edited entries in example snapshot YAML files.
  • MR: Implement glfm_example_status.yml (!87847 - merged)
• Improve GLFM static HTML snapshots
  • MR: Improve GLFM static HTML snapshots (!89214 - merged)
• Add Markdown Snapshot spec for GLFM backend/static HTML examples
  • MR: Create spec/requests/api/markdown_snapshot_spec.rb (!89219 - merged)
• Add support for regex-ignoring variable values in static HTML in spec/fixtures/glfm/example_snapshots/html.yml. This is a port of the "substitutions" logic from Golden Master specs.
  • MR: Add substitutions to GLFM static HTML snapshot ... (!89301 - merged)
• Add Markdown Snapshot spec for GLFM frontend/WYSIWYG HTML examples
  • MR: Create spec/frontend/content_editor/markdown_sn... (!89953 - merged) (branch caw-glfm-frontend-example-snapshot-specs)
• Move everything from /spec/fixtures/glfm to under /glfm_specification - Enrique and Himanshi are in favor.
  • MR: Move GLFM example_snapshots under glfm_specific... (!90554 - merged) (branch caw-glfm-move-example-snapshots)
• Implement scripts/glfm/run-snapshot-tests.sh
  • MR: Add scripts/glfm/run-snapshot-tests.sh (!90555 - merged) (branch caw-glfm-create-run-snapshot-tests-script)
• Make fixes to reduce variability in example snapshots, in order to reduce the need to manually curate normalizations:
  • Introduce setting to override random behavior of footnote IDs (GITLAB_TEST_FOOTNOTE_ID ENV var)
    • MR: Allow override of random footnote ID (!90959 - merged) (branch caw-glfm-allow-override-of-random-footnote-id)
    • MR: Set test footnote ID for GLFM snapshot examples (!90992 - merged) (branch caw-glfm-set-test-footnote-id)
• Support GLFM example snapshots in third-level headings
  • MR: Add support for nesting GLFM examples in H3s (!95302 - merged) (branch caw-add-glfm-h3-example-support)
• Convert render_static_html.rb to use rspec, and add deterministic fixtures.
  • MR: Convert render_static_html.rb to rspec (!95437 - merged) (branch caw-convert-render-static-html-script-to-rspec)
• Refactor YAML access and symbols in GLFM scripts and tests
  • MR: A01. Refactor YAML access and symbols in GLFM s... (!96318 - merged) (branch caw-glfm-yaml-symbol-refactors)
• DRY up fixture creation between spec/support/shared_contexts/markdown_snapshot_shared_examples.rb and scripts/lib/glfm/render_static_html.rb. This is necessary to prevent failures when running spec/support/shared_contexts/markdown_snapshot_shared_examples.rb due to non-normalized values. This should also include documentation updates to the normalization section to indicate other means we use to avoid the need for normalization (i.e. fixtures, and GITLAB_TEST_FOOTNOTE_ID)
  • MR: A02. DRY up and clean up GLFM fixtures (!96325 - merged) (branch: caw-glfm-cleanup-fixtures)
• Add API request overrides support to GLFM example snapshot specs. This will allow examples which use different preview_markdown contexts, and also drive support for EE examples such as group wikis.
  • MR: A03. Add metadata support to GLFM example snaps... (!92507 - merged) (caw-glfm-add-examples-metadata)
• Raise an exception if any of the manually-curated config files (glfm_specification/input/gitlab_flavored_markdown/glfm_example_*.yml) contain references to example names which do not exist in glfm_specification/example_snapshots/examples_index.yml. This will catch errors when inserting new example header sections that are not at the last/end position, but not updating the manual files with the new renumbered example names.
  • MR: Add error checking for GLFM config files (!98735 - merged) (branch caw-glfm-error-checking-example-names)
• Rename intro.txt to intro.md and ghfm_spec_v_0.29.txt to ghfm_spec_v_0.29.md
  • MR: Rename GLFM input files to *.md (!100227 - merged) (caw-glfm-rename-md-files)
• Implement split between official specification and internal extensions as documented by Update GLFM docs for official specification vs ... (!98493 - merged)
  • MR: !100472 (merged) (caw-glfm-impl-split-official-vs-extensions)
• Add support for automatically generating spec.html when running update-specification.rb
  • MR: !100489 (merged) (caw-glfm-generate-spec-html)
• Setup new CI job which runs the update-specification.rb and update-example-snapshots.rb scripts, and fails if this results in any diffs to the generated and committed output specification files or example snapshot files. This job will ensure that all changes to the input specification files are reflected in those output files.
  • MR: !101118 (merged) (branch caw-glfm-ci)
  • Revisit CI job which was reverted due to problems with docs-only and FOSS-only pipelines: Remove .gitlab-ci.yml for GLFM verification script (!101453 - merged)
    • MR: !101589 (merged) (branch caw-glfm-ci-2)
• Refactor/rename input files as discussed and agreed in this thread: #375300 (comment 1131831270).
  • MR: Reorganize GLFM input and output file contents (!101413 - merged) (branch caw-glfm-reorg-input-files)
• DRY up duplication with import statements and createTestEditor extensions list between spec/frontend/content_editor/render_html_and_json_for_all_examples.js and spec/frontend/content_editor/services/markdown_serializer_spec.js
  • MR: DRY up GLFM tiptapEditor helper (!101868 - merged) (branch caw-glfm-dry-up-js-extensions)
• TODO: Look into possibility of adding CI rules to force spec/frontend/content_editor/markdown_snapshot_spec.js frontend spec to run when GLFM specification files change. See !102244 (comment 1150457127) for context. See also ENOENT: no such file or directory, open '/build... (#379080 - closed)
  • Use imports to solve this: !102244 (comment 1150751210) and !102244 (comment 1150753245)
    • MR: Run markdown_snapshot_spec.js when needed (!102688 - merged) (branch caw-glfm-ci-triggers)
  • Also fix for backend:
    • MR: Run markdown_snapshot_spec.rb when needed (!102693 - merged) (branch caw-glfm-ci-triggers-2)
• Perform static analysis to enforce that every existing backend and frontend example has at least one corresponding example in the spec. See related comment
  • Need to decide exactly what this means, and if it is even possible.
  • glfm_specification/output/glfm_snapshot_spec.md already contains all defined markdown examples. Do we want to try to somehow look at the existing code, specs, or user-facing markdown docs to try to determine if any examples are missing? How would we even do that?
  • UPDATE: This doesn't seem feasible/practical. Marking this task as closed without implementation.
• Backfill all missing GLFM examples from existing Golden Master examples, with appropriate glfm_example_status.yml entries for unimplemented/unsupported functionality (This is being tracked in a separate issue: Tracking Issue: Migrate existing specs Golden M... (#361242 - closed))
• Update spec.html and snapshot_spec.html to use the similar standard formatting, e.g. example numbering, as the official CommonMark/GHFM spec.html. This will require simulating the formatting of the Lua script and template from the CommonMark project: https://github.com/commonmark/commonmark-spec/blob/master/Makefile#L11. See also this related comment: !98381 (comment 1158434632)
  1. Part 1 - split examples - MR: GLFM HTML formatting Part 1: Split GLFM example... (!102822 - merged) (branch caw-glfm-format-html-part-1)
  2. Part 2 - styling, numbering, and table of contents - MR: GLFM HTML formatting Part 2: Styling, numbering... (!103094 - merged) (branch caw-glfm-format-html-part-2)
  3. See follow-on tasks below in "Other cleanup tasks" section below
• Review brainstorming document from Himanshu and Chad, and update tasks on this and other issues to reflect and flesh out that new planned direction around the parallel support for the "remark" vs "hybrid" deserializers: https://docs.google.com/document/d/1HUQKQFXYD5r4Rs7bPx96b3z5MpnfNhQNhuVhEBnUe-A/edit
• Implement support for running Markdown Conformance Testing for GLFM implementation against GHFM/CommonMark specification and GLFM spec.txt.
  • MR: TODO: (branch TODO)
• By default, any errors from rendering during scripts/glfm/update-example-snapshots.rb should fail the script. Require a --allow-unskipped-errors option to be passed, which will have existing behavior of including Error - check implementation as rendered output. The failure output should include the error message, as well as a pointer to the documentation on how to add an entry to skip the example to glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml
• Changes to example snapshot files should run relevant frontend and backend specs. See https://gitlab.com/gitlab-org/gitlab/-/pipelines/707254208 pipeline in Convert GLFM examples from old GM approach (!104933 - merged) for an example of them not getting run when they should.
• Make update-example-snapshots.rb check for any git diffs to snapshot files after running, and if there are any, print out a big red warning saying to verify all the changes are exactly as expected, and if not, to fix the problem or add appropriate entries to `glfm_example_status.yml. See related discussion in !106733 (comment 1209495621)
• Add a placeholder script for run-spec-tests.sh which just prints out a "NOT YET IMPLEMENTED" message with some explanatory text.
  • MR: Add placeholder run-spec-tests.sh (!107768 - merged)
• Decide where and how we will host spec.html and snapshot_spec.html. Followup to this thread: !84220 (comment 909137776)
  • MR: TODO: (branch TODO)

Other Documentation Tasks

• Update GLFM docs to fix flowchart (!86528 - merged)
• Update GLFM docs with new headers (!86604 - merged)
• Add links in GLFM dev guide docs to source for ... (!90562 - merged)
• Make a diagram documenting hierarchy and terminology of various markdown specifications.
  • Update GLFM Specification Guide with Markdown S... (!93376 - merged)
• Add a "Summary" section: Add summary to GLFM guide (!98273 - merged)
• Update documentation to distinguish between the GLFM official specification and GLFM internal extensions. This is a follow-up to plans outlined in these threads:
  • !93376 (comment 1045183892)
  • !94985 (comment 1069662953)
  • !94985 (comment 1089744937)
  • MR: Update GLFM docs for official specification vs ... (!98493 - merged)
• Fix broken link. See !85982 (comment 1031524242)
  • MR: Fix incorrect URL in GLFM guide (!99072 - merged)
• Update https://docs.gitlab.com/ee/user/markdown.html to add a mention of and link to https://docs.gitlab.com/ee/development/gitlab_flavored_markdown/specification_guide/
  • MR: Add a link to GLFM spec guide (!99073 - merged)
• Add/update docs around input specification config files, to reflect changes made in Add error checking for GLFM config files (!98735 - merged)
  • Add a new H4 heading after https://docs.gitlab.com/ee/development/gitlab_flavored_markdown/specification_guide/#input-specification-files named Input Specification Config Files which contains glfm_example_*.yml files.
  • MR: Reorganize GLFM input specification files section (!99078 - merged)
• Clearly indicate non-implemented parts (e.g. conformance testing). See !85982 (comment 1031528958)
  • MR: Add GLFM note on conformance tests (!99081 - merged)
• Add docs for Add error checking for GLFM config files (!98735 - merged), including around special case for Input Specification Config Files where examples starting with 00_ are skipped for validation, and can be used for YAML anchors.
  • MR: Add docs on error handling for GLFM input speci... (!99083 - merged), Branch: caw-glfm-add-note-on-00-example-naming
• Fix snapshot_spec filename in GLFM docs (!102825 - merged)
• Add note about run-spec-tests.sh not yet being implemented, and only showing a placeholder message.
  • MR: Add note to GLFM docs on unimplemented script (!107775 - merged)
• Remove obsolete note about spec.html:
  • MR: Remove obsolete note from GLFM docs (!107771 - merged)
• TODO: Switch to shortcut reference links as suggested in this comment
• TODO: Document usage and details of snapshot test spec files:
  • Update of section on scripts/glfm/run-snapshot-tests.sh
  • Add details on CE and EE versions of spec/requests/api/markdown_snapshot_spec.rb
  • Add details on CE and EE versions of spec/frontend/content_editor/markdown_snapshot_spec.js
  • document usage of FOCUSED_MARKDOWN_EXAMPLES
• TODO: Add documentation around details and requirements of spec.txt header and disabled example parsing that are currently only documented in code/tests in scripts/lib/glfm/parse_examples.rb and scripts/lib/glfm/update_example_snapshots.rb#add_example_names. See discussion in these threads: !90561 (comment 999901748) and !90561 (comment 1004779132)
• TODO: Add some text to https://docs.gitlab.com/ee/development/gitlab_flavored_markdown/specification_guide/#markdown-snapshot-testing which explains how our snapshot testing is acting as end-to-end "top of pyramid" integration testing.
  • MR: Add snapshot testing note to GLFM docs (!107774 - merged)
• TODO: Make a doc note that only the pre-deserialization, non-dynamic frontend behavior is being tested.
• TODO: Update workflow sections with more details (based on feedback from Enrique)
  • Add initial instructions on how to add an example
  • Make clear that conformance testing is not yet implemented.
  • Change split of two workflows. Instead have one workflow for adding examples for already-implemented functionality, and another for adding examples and status entries for TDD-ing functionality that does not yet exist
  • Document meaning of tokens and extensions, and special gitlab token in backtick-delimited example format
  • Explain how to make a canonical example. It should be the minimal representation, which is a common subset of the static and wysiwyg HTML that can be derived via normalization. Also document that there are some cases where this may not be possible, such as PlantUML diagrams.
• TODO: Add a mermaid diagram to docs section for verify-all-generated-files-are-up-to-date.rb (flowchart showing success/failure cases).
• TODO: Add note to docs section on various specifications about GHFM HTML version being outdated, with link to issue: https://github.com/github/cmark-gfm/issues/288
  • MR: Add note about GHFM HTML spec being outdated (!107772 - merged)
• TODO: Revisit terminology of dialect vs. flavor. Follow-up from this thread: !84220 (comment 903091005)
• TODO: Add some notes on how and where to update example numberings if you reorder examples in the input markdown files. Use this comment's explanation as a starting point: !106733 (comment 1206573879)
• TODO: Update/add workflow docs with these details: !106733 (comment 1206725109)
• TODO: Update/add some docs (maybe workflow section, maybe other sections) with these explanations of deserialization and related topics: !106733 (comment 1209495621)
• TODO: Update text Goals section of the Specification Guide and the output specification files section to reflect the fact that it's not just spec.txt anymore, since that no longer contains CommonMark+GFM examples.
• Update diagram and text in https://docs.gitlab.com/ee/development/gitlab_flavored_markdown/specification_guide/#update-specificationrb-script to reflect the fact that spec.txt only contains the GLFM official specification, and no longer contains CommonMark+GFM examples or internal extensions.
• Update https://docs.gitlab.com/ee/development/gitlab_flavored_markdown/specification_guide to fix outdated references and charts for scripts and filenames.
  • MR: Fix outdated references in GLFM docs - Pass 1 (!107777 - merged)

Other Cleanup Tasks

• Support full diffs in fast_spec_helper (!89205 - merged)
• Fix roulette for GLFM scripts (!89222 - merged)
• Fix GLFM typos and lint/inspection warnings (!90560 - merged)
• Fix bug in GLFM spec handling (!90561 - merged)
• Simplify headers logic in parse_examples (!90993 - merged)
• Uncomment jest in markdown_snapshot_spec.rb (!92504 - merged)
• Add some clarification to GLFM fixture (!94905 - merged)
• Remove unnecessary footnote normalization from ... (!95537 - merged)
• Remove unused GLFM example extension annotations (!95761 - merged)
• Minor cleanups to GLFM update_example_snapshots... (!95765 - merged)
• Fix typo in scripts/lib/glfm/render_static_html.rb (!100474 - merged)
• Switch all manual ENV stubs to use stub_env helper: Switch to stub_env in update_specification_spec.rb (!101870 - merged)
• Improve output and usefulness of glfm-verify job (!103130 - merged)
• TODO: Extract javascript out of scripts/lib/glfm/specification_html_template.erb and add unit tests for it.
• TODO: Add formatting to spaces within examples. See TODO comment in CSS in scripts/lib/glfm/specification_html_template.erb.
• TODO: Refactor parse_examples.rb to be cleaner and more idiomatic Ru by than original straight port of Python code. Goal: Remove need for rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity, Metrics/AbcSize
• TODO: Add "This file was generated, do not edit!" to the top of all generated files
• TODO: DRY up duplication in pattern for raising error if files are empty. See review comment here: !100472 (comment 1135764904). Completed as part of Reorganize GLFM input and output file contents (!101413 - merged)
• TODO: Clean up warnings in JetBrains IDEs and add docs. See start in branch caw-glfm-jetbrains-ide-warnings-cleaup
• Remove unnecessary CE reference in spec/frontend/content_editor/markdown_snapshot_spec.js (fixed in Run markdown_snapshot_spec.js when needed (!102688 - merged))
• Inline spec/frontend/content_editor/markdown_snapshot_spec_helper.js into spec/frontend/content_editor/markdown_snapshot_spec.js, because there no longer needs to be a CE vs. EE split on the frontend.
  • MR: Refactor inline markdown snapshot helper spec (!107767 - merged)
• TODO: Revisit usage of scripts/lib directory, because it sometimes isn't handled properly by Spring. Possibly needs to be added to Spring config. Follow up to this thread: !84220 (comment 912060119)

Third Party Tasks

• GitHub Flavored Markdown HTML-rendered spec is out of sync with their latest spec.txt. See issue: https://github.com/github/cmark-gfm/issues/288. See also slack thread.

Current MR Rebase Branch Queue

⚠ ⚠ ⚠ DON'T FORGET TO UPDATE COMMIT MESSAGES WITH MR LINKS ⚠ ⚠ ⚠

1. placeholder-upstream-1
2. git co placeholder-upstream-2 && git rebase head~1 --onto placeholder-upstream-1