Skip to content

Support running GitLab Markdown Golden Master tests against standard GitHub+CommonMark spec.txt

Chad Woolley requested to merge caw-markdown-specs into master

What does this MR do and why?

Related: #345726 (closed)

Provides a process to run the existing backend and frontend Golden Master Markdown testing framework against the standard GitHub-Flavored Markdown spec.txt as specified in https://github.com/github/cmark-gfm/blob/master/test/spec.txt (which is a strict superset of the CommonMark spec).

NOTE: Currently only supported for backend, frontend tests in

Overview of process:

  1. Run scripts/markdown/convert_spec_txt_to_golden_master_examples_yml.rb to generate the YML file (see file comments for description and usage notes). NOTE: You don't have to do this step, the generated file is already checked in.
  2. Run backend spec against the generated file: spec/requests/api/markdown_github_golden_master_spec.rb
  3. Run frontend spec against the generated file: spec/frontend/content_editor/markdown_github_processing_spec.js

Both specs currently fail on most of the examples due to differences in GitLab's markdown rendering from the original spec.txt spec.

TODO

  • Question: Why are there more tests (674 backend, 673 frontend) being run than there are examples in the spec.txt (649) 🤔 ???

Initial commit run spec compliance

  1. For frontend, 73 out of 673 examples pass. NOTE: These failures are due to HTML-to-Markdown rendering differences, e.g. rendering headers with leading # instead of a line of --- underneath.
  2. For backend, 287 of 674 examples pass. NOTE: Some of these may still be false negatives because we need to strip out GitLab-specific metadata attributes such as id, class, href, aria-hidden - I've already got a method in place to strip these out, I just need to identify what they all are. However, others seem legit, like self-closing tag differences, etc.).

Latest commit run spec compliance

(may not be up-to-date)

  1. For backend, 382 of 674 examples now passing

Failures remaining to address

We should look at failures caused by these, and either fix them in the markdown rendering, or else make the spec ignore them:

  1. Self-closing tags
  2. base64-encoded <img> data
  3. <img> tag wrapped in <a> tag instead of just bare <img> tag
  4. Wrapping HTML in <div>\n - NOTE: Some are already ignored as metadata
  5. Other extra <div> tags in HTML (which may have only had ignored metadata attributes)
  6. Unnecessary extra <span> tags - NOTE: Some are already ignored as metadata
  7. ???

Screenshots or screen recordings

Example backend results:

backend test run

Example frontend results:

frontend test run

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Chad Woolley

Merge request reports