Spike - add tests to the .gitlab-ci.json schema
This issue was open following the discussion in #344894 (closed)
Gitlab now hosts an officially available version of the json schema. The base schema was imported from schemastore (see #218473 (closed)) and additional missing keywords have been, and will continue to be, added to the schema over time.
The schema is being used in the pipeline editor to validate
.gitlab-ci.yml
files. Since we will be updating the schema regularly, we should look into adding tests to make sure the schema is correct.
The purpose of this spike issue to understand what are the required test and come up with a solid plan for #344894 (closed)
How to test json schemas
(From #344894 (comment 725003543))
Basing this on how Schemastore tests json schemas, the general process is as follows. We can provide examples of correct and incorrect snippets of json code, which we run against the definitions we created in the json schema using a json validator. Of course, positive tests should pass and negative tests (incorrect json snippets) are expected to fail against these definitions.
How does schemastore do it?
It looks like schemastore uses ajv as its json validator. The actual testing is scripted in their Gruntfile.
📝 Proposal
Per the remark in this github comment:
Remark: GitLab should copy the two GitLab test file directories. I don't see it there. GitLab should have a JSON schema validator on the server. Perhaps a validator is already active, but without using test files.
Schemastore already has test files to validate some keywords in the schema when they were still hosting the schema file:
(Note that these files have been deleted already on the main branch, so I am linking to the last commit when they were available).
As a first step, we can import these schemastore test files first and write a much more simple script to run these against out json schema using ajv. The goal is to confirm that the schema definitions validate different shapes of data correctly.
We can do a POC and explore this idea first to see how we can set this up for gitlab and if this is the direction we want to go, or if we should explore other ways of testing.
Note: Our schema is used for validating .gitlab-ci.yml
which is in yaml. You might notice that the test files are in json format now (which makes sense, since we are using a json schema). Maybe we can explore using yaml for our test files and include yaml-to-json conversion in testing, but that's something I think we can explore later on when we've established how we'll test our schema.
❓ Concerns
Some concerns to bring up with this proposal are:
- This requires a new dependency (the json validator) for testing
- Writing the script to run the test files tests against the json schema (and adding the script to our testing pipeline?). This is also outside of our usual testing practices since we're introducing a new process just for testing json schemas, so I'm not sure if there are other considerations to look into about this
- Maybe there's a simpler way to test them?