Introduce google/go-cmp to compare test results (!25) · Merge requests · GitLab.org / security-products / analyzers / secrets

Takuya Noguchi requested to merge improve-tests-with-comparer into master Feb 20, 2020

What does this MR do?

Introduces google/go-cmp to compare test results by replacing reflect.DeepEqual function.

This would improve contributor experience when they encounters test failures, e.g.) https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/jobs/443092932

What to get?

Before

 --- FAIL: TestToIssues (0.01s)
     gitleaks_test.go:341: Wrong result. Expected:
         []issue.Issue{issue.Issue{Category:"sast", Name:"RSA private key", Message:"RSA private key", Description:"RSA private key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/id_rsa:8bcac7908eb950419537b91e19adc83ce2c9cbfdacf4f81157fdadfec11f7017:RSA", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/id_rsa", LineStart:1, LineEnd:15, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID RSA", Value:"RSA", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"AWS API key", Message:"AWS API key", Description:"Amazon Web Services API key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/main.go:8a22accb113d641b78b389ccce92fca96acbe2fd4f1701ead6783768fdbe9d8a:AWS", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:7, LineEnd:7, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID AWS", Value:"AWS", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"High entropy string", Message:"High entropy string", Description:"A string with high entropy was found, this could be a secret", CompareKey:"testdata/main.go:4bc941a5c41c5460ab3d1895453e66d7032ae11dd401154d396f906d4dda0add:Entropy", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:9, LineEnd:9, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID Entropy", Value:"Entropy", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"RSA private key", Message:"RSA private key", Description:"RSA private key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/main.go:430e004686f8f9c2d11ce84da58bd94d1fceb70b0296e46dd8e1ca059ebf7e92:RSA", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:11, LineEnd:40, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID RSA", Value:"RSA", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Stripe", Message:"Stripe", Description:"Stripe API key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/main.go:0b16c77410a5918254c1b2bdc7576b87d9ffc7dcedf29b7bdca0b423f6209009:Stripe", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:50, LineEnd:50, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID Stripe", Value:"Stripe", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Generic API Key", Message:"Generic API Key", Description:"Unknown API key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/main.go:9ac057c201cda016677f2ddeb03d4c59991007a13b9e07917d1f8782e9564970:Generic API Key", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:52, LineEnd:52, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID Generic API Key", Value:"Generic API Key", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Generic API Key", Message:"Generic API Key", Description:"Unknown API key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/main.go:f4ac6d9fd61b258a20a5d9c5aea0e0b48abc60d50edd149021d875066e6c592b:Generic API Key", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:53, LineEnd:53, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID Generic API Key", Value:"Generic API Key", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Generic API Key", Message:"Generic API Key", Description:"Unknown API key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/main.go:72c5f23ed973b0cc312a2c05cae2798ec924f82e3154c86af53e9db0101ec22c:Generic API Key", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:54, LineEnd:54, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID Generic API Key", Value:"Generic API Key", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Generic API Key", Message:"Generic API Key", Description:"Unknown API key detected; please remove and revoke it if this is a leak.", CompareKey:"testdata/main.go:ecd55deb344afa69c181ae430c0352de88dd565ab7ed454535824d62f18882fd:Generic API Key", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:55, LineEnd:55, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID Generic API Key", Value:"Generic API Key", URL:""}}, Links:[]issue.Link(nil)}}
         but got:
         []issue.Issue{issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/id_rsa:8bcac7908eb950419537b91e19adc83ce2c9cbfdacf4f81157fdadfec11f7017:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/id_rsa", LineStart:1, LineEnd:15, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:8a22accb113d641b78b389ccce92fca96acbe2fd4f1701ead6783768fdbe9d8a:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:7, LineEnd:7, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:4bc941a5c41c5460ab3d1895453e66d7032ae11dd401154d396f906d4dda0add:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:9, LineEnd:9, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:430e004686f8f9c2d11ce84da58bd94d1fceb70b0296e46dd8e1ca059ebf7e92:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:11, LineEnd:40, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:0b16c77410a5918254c1b2bdc7576b87d9ffc7dcedf29b7bdca0b423f6209009:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:50, LineEnd:50, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:9ac057c201cda016677f2ddeb03d4c59991007a13b9e07917d1f8782e9564970:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:52, LineEnd:52, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:f4ac6d9fd61b258a20a5d9c5aea0e0b48abc60d50edd149021d875066e6c592b:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:53, LineEnd:53, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:72c5f23ed973b0cc312a2c05cae2798ec924f82e3154c86af53e9db0101ec22c:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:54, LineEnd:54, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}, issue.Issue{Category:"sast", Name:"Gitleaks rule ", Message:"Gitleaks rule ", Description:"Gitleaks rule  detected a secret", CompareKey:"testdata/main.go:ecd55deb344afa69c181ae430c0352de88dd565ab7ed454535824d62f18882fd:", Severity:7, Confidence:2, Solution:"", Scanner:issue.Scanner{ID:"gitleaks", Name:"Gitleaks"}, Location:issue.Location{File:"testdata/main.go", LineStart:55, LineEnd:55, Class:"", Method:"", Dependency:issue.Dependency{Package:issue.Package{Name:""}, Version:""}}, Identifiers:[]issue.Identifier{issue.Identifier{Type:"gitleaks_rule_id", Name:"Gitleaks rule ID ", Value:"", URL:""}}, Links:[]issue.Link(nil)}}
 FAIL

After

=== RUN   TestToIssues
--- FAIL: TestToIssues (0.00s)
    gitleaks_test.go:342: toIssues() results mismatch (-want +got):
          []issue.Issue{
          	{
          		... // 8 identical fields
          		Scanner:  issue.Scanner{ID: "gitleaks", Name: "Gitleaks"},
          		Location: issue.Location{File: "testdata/id_rsa", LineStart: 1, LineEnd: 15},
          		Identifiers: []issue.Identifier{
          			{
          				Type:  "gitleaks_rule_id",
        - 				Name:  "Gitleaks rule ID test RSA",
        + 				Name:  "Gitleaks rule ID RSA",
          				Value: "RSA",
          				URL:   "",
          			},
          		},
          		Links: nil,
          	},
          	{Category: "sast", Name: "AWS API key", Message: "AWS API key", Description: "Amazon Web Services API key detected; please remove and revoke it if this is a leak.", CompareKey: "testdata/main.go:8a22accb113d641b78b389ccce92fca96acbe2fd4f1701ead6783768fdbe9d8a:AWS", Severity: s"Critical", Confidence: s"Unknown", Scanner: issue.Scanner{ID: "gitleaks", Name: "Gitleaks"}, Location: issue.Location{File: "testdata/main.go", LineStart: 7, LineEnd: 7}, Identifiers: []issue.Identifier{{Type: "gitleaks_rule_id", Name: "Gitleaks rule ID AWS", Value: "AWS"}}},
          	{Category: "sast", Name: "High entropy string", Message: "High entropy string", Description: "A string with high entropy was found, this could be a secret", CompareKey: "testdata/main.go:4bc941a5c41c5460ab3d1895453e66d7032ae11dd401154d396f906d4dda0add:Entropy", Severity: s"Critical", Confidence: s"Unknown", Scanner: issue.Scanner{ID: "gitleaks", Name: "Gitleaks"}, Location: issue.Location{File: "testdata/main.go", LineStart: 9, LineEnd: 9}, Identifiers: []issue.Identifier{{Type: "gitleaks_rule_id", Name: "Gitleaks rule ID Entropy", Value: "Entropy"}}},
          	... // 3 identical elements
          	{Category: "sast", Name: "Generic API Key", Message: "Generic API Key", Description: "Unknown API key detected; please remove and revoke it if this is a leak.", CompareKey: "testdata/main.go:f4ac6d9fd61b258a20a5d9c5aea0e0b48abc60d50edd149021d875066e6c592b:Generic API Key", Severity: s"Critical", Confidence: s"Unknown", Scanner: issue.Scanner{ID: "gitleaks", Name: "Gitleaks"}, Location: issue.Location{File: "testdata/main.go", LineStart: 53, LineEnd: 53}, Identifiers: []issue.Identifier{{Type: "gitleaks_rule_id", Name: "Gitleaks rule ID Generic API Key", Value: "Generic API Key"}}},
          	{Category: "sast", Name: "Generic API Key", Message: "Generic API Key", Description: "Unknown API key detected; please remove and revoke it if this is a leak.", CompareKey: "testdata/main.go:72c5f23ed973b0cc312a2c05cae2798ec924f82e3154c86af53e9db0101ec22c:Generic API Key", Severity: s"Critical", Confidence: s"Unknown", Scanner: issue.Scanner{ID: "gitleaks", Name: "Gitleaks"}, Location: issue.Location{File: "testdata/main.go", LineStart: 54, LineEnd: 54}, Identifiers: []issue.Identifier{{Type: "gitleaks_rule_id", Name: "Gitleaks rule ID Generic API Key", Value: "Generic API Key"}}},
          	{
          		Category: "sast",
          		Name:     "Generic API Key",
          		Message:  "Generic API Key",
          		Description: strings.Join({
          			"U",
        - 			"NKNOWN",
        + 			"nknown",
          			" API key detected; please remove and revoke it if this is a leak",
          			".",
          		}, ""),
          		CompareKey: "testdata/main.go:ecd55deb344afa69c181ae430c0352de88dd565ab7ed454535824d62f18882fd:Generic API Key",
          		Severity:   s"Critical",
          		... // 6 identical fields
          	},
          }
FAIL

What are the relevant issue numbers?

n/a

Does this MR meet the acceptance criteria?

Conforms to the code review guidelines
Conforms to the Go guidelines
[n/a] Security reports checked/validated by reviewer

Edited Feb 20, 2020 by Takuya Noguchi

Introduce google/go-cmp to compare test results