Improve JSON validation error output
What does this MR do?
While working on Show one `shortest path` on the finding details... (gitlab-org/gitlab#348532 - closed) I ran into JSON validation errors because some required fields in the dependency scanning report were empty, however, it was impossible to figure out which fields were causing the errors, because the JSON validation error message includes the entire dependency-scanning-report-format.json schema file, which is over 2300 lines long:
Current JSON validation error
Failures:
1) report shared examples with Dependency Scanning report and the report is invalid behaves like valid report passes schema validation without errors
Failure/Error:
expect(JSONSchemer.schema(schema).validate(report).to_a
).to eql([])
expected: []
got: [{"data"=>"", "data_pointer"=>"/scan/analyzer/id", "root_schema"=>{"$schema"=>"http://json-schema.org..., "schema_pointer"=>"/properties/scan/properties/analyzer/properties/version", "type"=>"minLength"}]
(compared using eql?)
Diff:
@@ -1,2382 +1,4763 @@
-[]
+[{"data"=>"",
+ "data_pointer"=>"/scan/analyzer/id",
+ "root_schema"=>
+ {"$schema"=>"http://json-schema.org/draft-07/schema#",
+ "additionalProperties"=>true,
+ "definitions"=>
+ {"code"=>
+ {"description"=>"A codeblock",
+ "properties"=>
+ {"lang"=>{"description"=>"A programming language", "type"=>"string"},
+ "type"=>{"const"=>"code"},
+ "value"=>{"type"=>"string"}},
+ "required"=>["type", "value"],
+ "type"=>"object"},
+ "commit"=>
+ {"description"=>"A commit/tag/branch within the GitLab project",
+ "properties"=>
+ {"type"=>{"const"=>"commit"},
+ "value"=>
+ {"description"=>"The commit SHA",
+ "minLength"=>1,
+ "type"=>"string"}},
+ "required"=>["type", "value"],
+ "type"=>"object"},
+ "detail_type"=>
+ {"oneOf"=>
+ [{"$ref"=>"#/definitions/named_list"},
+ {"$ref"=>"#/definitions/list"},
+ {"$ref"=>"#/definitions/table"},
+ {"$ref"=>"#/definitions/text"},
+ {"$ref"=>"#/definitions/url"},
+ {"$ref"=>"#/definitions/code"},
+ {"$ref"=>"#/definitions/value"},
+ {"$ref"=>"#/definitions/diff"},
+ {"$ref"=>"#/definitions/markdown"},
+ {"$ref"=>"#/definitions/commit"},
+ {"$ref"=>"#/definitions/file_location"},
+ {"$ref"=>"#/definitions/module_location"}]},
+ "diff"=>
+ {"description"=>"A diff",
+ "properties"=>
+ {"after"=>{"type"=>"string"},
+ "before"=>{"type"=>"string"},
+ "type"=>{"const"=>"diff"}},
+ "required"=>["type", "before", "after"],
+ "type"=>"object"},
+ "file_location"=>
+ {"description"=>"A location within a file in the project",
+ "properties"=>
+ {"file_name"=>{"minLength"=>1, "type"=>"string"},
+ "line_end"=>{"type"=>"integer"},
+ "line_start"=>{"type"=>"integer"},
+ "type"=>{"const"=>"file-location"}},
+ "required"=>["type", "file_name", "line_start"],
+ "type"=>"object"},
+ "list"=>
+ {"description"=>"A list of typed fields",
+ "properties"=>
+ {"items"=>
+ {"items"=>{"$ref"=>"#/definitions/detail_type"}, "type"=>"array"},
+ "type"=>{"const"=>"list"}},
+ "required"=>["type", "items"],
+ "type"=>"object"},
<snip>
+ "vulnerabilities"=>
+ {"description"=>"Array of vulnerability objects.",
+ "required"=>["dependency_files", "version", "vulnerabilities"],
+ "self"=>{"version"=>"14.0.4"},
+ "title"=>"Report format for GitLab Dependency Scanning"},
+ "schema"=>
+ {"description"=>"The version of the analyzer.",
+ "examples"=>["1.0.2"],
+ "minLength"=>1,
+ "type"=>"string"},
+ "schema_pointer"=>"/properties/scan/properties/analyzer/properties/version",
+ "type"=>"minLength"}]
Shared Example Group: "valid report" called from ./spec/report_shared_examples_spec.rb:23
# ./lib/gitlab_secure/integration_test/shared_examples/report_shared_examples.rb:152:in `block (2 levels) in <top (required)>'
Finished in 0.14811 seconds (files took 0.21104 seconds to load)
34 examples, 1 failure
Failed examples:
rspec ./spec/report_shared_examples_spec.rb:23 # report shared examples with Dependency Scanning report and the report is invalid behaves like valid report passes schema validation without errorsThis MR updates the JSON validation code to output the exact error that caused the failure. For example, instead of the above output, we now have:
1) report shared examples with Dependency Scanning report and the report is invalid behaves like valid report passes schema validation without errors
Failure/Error:
expect(validation_result.to_a).to be_empty, %|JSON validation against schema '#{schema_path}' failed with the following errors:\n\n| +
%| #{validation_errors.join("\n ")}|
JSON validation against schema 'security-report-schemas/v14.0.4/dependency-scanning-report-format.json' failed with the following errors:
property '/scan/analyzer/id' is invalid: error_type=minLength
property '/scan/analyzer/name' is invalid: error_type=minLength
property '/scan/analyzer/vendor/name' is invalid: error_type=minLength
property '/scan/analyzer/version' is invalid: error_type=minLength
Shared Example Group: "valid report" called from ./spec/report_shared_examples_spec.rb:23
# ./lib/gitlab_secure/integration_test/shared_examples/report_shared_examples.rb:150:in `block (2 levels) in <top (required)>'The above error message now makes it clear exactly which fields are causing the JSON validation to fail.
What are the relevant issue numbers?
gitlab-org/gitlab#348532 (closed)
Testing
- Tested against gemnasium, image tests pass as expected.
- Output of JSON validation failures before this change, validation error messages are extremely long.
- Output of JSON validation failures after this change, validation error messages are short and show the exact field that caused the failure.
Edited by Adam Cohen