Draft: POC to add tests for json schemas

What does this MR do and why?

POC for this spike: #345238 (closed)

This creates a base test helper for testing json schemas using ajv. The test helper allows testing with json files and yaml files.

Spike Breakdown

This spike has two goals:

• Import existing tests for gitlab-ci.json from schemastore and set up a testing suite for them.
• See if we can use yaml test files instead of json test files. Users will be writing their CI configurations in yaml, so this format is much closer to the input that we would like to test.

Here is the breakdown of the POC:

Commit	Description
!79521 (44e0f760)	Create spec directory with tests imported from schemastore
!79521 (adb25785)	Add ajv packages as dev dependencies
!79521 (c36445fc)	Create sample script to test CI json schema
!79521 (fe02fee6)	Fix incorrect json tests
!79521 (7089926b)	Add missing tests in yaml format
!79521 (78c3389d)	Test json schema using yaml tests

Yarn packages

I've added ajv and ajv-formats. ajv is what we can use to validate json schemas using json test files. This is also the same package used by schemastore to test its json schemas.

ajv-formats defines different formats (such as URI, date-time, etc) so they can be used in json schema definitions and validations.

Json Tests

The json tests are all imported from schemastore:

• Test Files
• Negative Tests Files

These files have been deleted already on the main branch, so I am linking to the last commit when they were available. We've also been given permission (recommendation) to copy the test files to GitLab.

I left these files mostly unchanged, except for a few corrections to gitlab-ci.json and retry.json since these files had outdated tests.

YAML tests

We've added new keywords to the CI json schema since we started hosting it on GitLab. These new definitions don't have json tests yet, so I used this as an opportunity to see if we can create tests in yaml format instead.

I've added positive and negative tests for the following:

• cache:when
• null filter
• include:rules
• trigger:include
• workflow:rules:changes
• workflow:rules:exists

Refer to the following MR changes:

• !74164 (merged)
• !70779 (merged)
• !79335 (merged)

How It Works

This is how the spec/frontend/editor/schema directory should look like now:

spec/frontend/editor/schema
├── base_test_setup.js
└── ci
    ├── ci_schema_spec.js
    ├── json_tests
    │   ├── negative_tests
    │   └── positive_tests
    └── yaml_tests
        ├── negative_tests
        └── positive_tests

Positive tests are json/yaml snippets that we expect to be valid. Negative tests are json/yaml snippets that we expect to be invalid (i.e. they intentionally use the schema incorrectly). This allows us the check that our json schema correctly validates different examples of input.

base_test_setup.js is our test helper and initializes ajv for us. The actual tests are in ci_schema_spec.js. What it does is that it loops through all the files in the test directories and checks with ajv if the json schema of our choosing validates them.

To help make writing these tests easier, I extended jest::expect with the new methods toValidateJsonSchema and toInvalidateJsonSchema. These methods will parse ajv's error messages to check if the file is valid or not.

Thus, we can write our tests this way:

expect(positiveFileToTest).toValidateJsonSchema(schema);
expect(negativeFileToTest).toInvalidateJsonSchema(schema);

With this setup, developers and contributors who want to update the CI schema will only need to add or update the yaml/json test files in the test directories, and then run the script to see if it works. No need to fiddle with the code, they just need to provide the input to test. 😄

Screenshots or screen recordings

Tests Pass	Tests Fail

How to set up and validate locally

1. Install missing yarn packages.
2. Run the following command:

yarn jest --watch spec/frontend/editor/schema/ci/ci_schema_spec.js

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.