Security report warnings that will become errors in 15.0

This issue is being used to collate all the security report warnings that should be fixed before %15.0 lest they become errors and prevent the reports from being ingested.

Kibana dashboard: https://log.gprd.gitlab.net/app/dashboards#/view/dbdedff0-b53a-11ec-b73f-692cc1ae8214?_g=h@3a04046

added to epic &6968 (closed)

added all Sec groups sectionsec labels

Via @plafoucriere (thanks!): https://gitlab.com/gitlab-com/www-gitlab-com/-/pipelines/494189051/security

• bundler-audit-dependency_scanning
  • [Schema] root is missing required keys: dependency_files

Being tracked in: #356259 (closed)

@thiagocsf We have more warnings today:

@plafoucriere, is that still in https://gitlab.com/gitlab-com/www-gitlab-com/-/pipelines?

@thiagocsf this is in https://gitlab.com/gitlab-org/gitlab this time.

Example pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/513565624/security

@gonzoyumo, are any of these scanners maintained by groupcomposition analysis?

@dappelt, IIRC you were involved with package hunter. It looks like it's using a very old version of the schema, and GitLab %15.0 will reject those reports. If I create an issue to handle this, do you know who should I assign it to?

@NicoleSchwartz, it seems that #356259 (closed) addressed gitlab-depscan, but the same "missing required keys: dependency_files" warning is being issued for:

• https://gitlab.com/gitlab-org/security-products/analyzers/retire.js (GitLab Retire.js analyzer v2.12.2)
• https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit (GitLab bundler-audit analyzer v2.11.8)
• https://gitlab.com/gitlab-org/security-products/analyzers/npm-audit (GitLab NPM audit analyzer v1.4.1)

Can I please leave this with you and @gonzoyumo?

@thiagocsf you may, however as of 15.0 we are removing reture.js and bundler-audit so at this time I choose to ignore those.

https://docs.gitlab.com/ee/update/deprecations.html#bundler-audit-dependency-scanning-tool

https://docs.gitlab.com/ee/update/deprecations.html#retire-js-dependency-scanning-tool

I will make an issue to investigate https://gitlab.com/gitlab-org/security-products/analyzers/npm-audit (GitLab NPM audit analyzer v1.4.1)

#358739 (closed)

@NicoleSchwartz that counts as "taken care of" in my book! Thanks

It looks like it's using a very old version of the schema, and GitLab %15.0 will reject those reports. If I create an issue to handle this, do you know who should I assign it to?

@thiagocsf feel free to assign it to me.

@thiagocsf @NicoleSchwartz I've created !84993 (merged) to remove bundler-audit and retire.js from the CI config of the GitLab project.

I'm not sure to understand this change @gonzoyumo: These 2 analyzers are coming from the official template so removing them from the gitlab-org/gitlab project will just silent the warnings there. Anyone else using dependency scanning and either ruby and/or html (cf rules) will have the same issue, right?

@plafoucriere as of 15.0 they are both removed so we want people to get warned/errors at that point

@NicoleSchwartz Thanks! I missed that. That makes sense

Bundler-audit and Retire.Js have also been removed from the handbook project: gitlab-com/www-gitlab-com!102483 (merged)

changed the description

changed milestone to %15.0

added backend featureenhancement groupthreat insights spike labels

added typefeature label

assigned to @thiagocsf

@Quintasan, once we have logging, can you please post a link to the kibana query here?

Sure, will do that. I expect to have the logging MR ready today/tomorrow.

Took longer than expected.

@thiagocsf I added some graphs to Threat Insights Engineering dashboard

A query would be https://log.gprd.gitlab.net/goto/bf0ab290-b53b-11ec-afaf-2bca15dfbf33

Setting label(s) devopssecure based on groupthreat insights.

added devopssecure label

changed the description

A Large Ultimate customer is presently encountering:

Error parsing security reports eslinst-sast(1): [IngestionError] Ingestion failed for some vulnerabilities

@thiagocsf Is that the kind of error that this issue is meant to capture?

This customer is on gitlab.com where VALIDATE_SCHEMA is deprecated. GitLab team members with access to Zendesk can learn more in the ticket.

Hi, @bcarranza. Thanks for the mention and the links.

TL;DR: the change in %15.0 does not affect this scenario.

Indeed VALIDATE_SCHEMA is now deprecated but, regardless of that, this setting doesn't affect whether or not the warnings are produced. These errors have likely always been happening, but since %14.9 we have visibility on them.

I've created a new project and provided the report from the Zendesk ticket as a security report artifact: https://gitlab.com/gitlab-org/secure/tests/test-for-issue-357038/-/pipelines/511939898/security

No errors were reported, which confirms that the error - at least for this report - is not due to schema validation. A previous report might still have contained invalid entries, but I don't know that for sure yet, otherwise I would hand this over to the groupstatic analysis team.

I also ran a pipeline with VALIDATE_SCHEMA: "true" and, as expected, no errors are displayed: https://gitlab.com/gitlab-org/secure/tests/test-for-issue-357038/-/pipelines/511947806.

Since the error we're seeing is not related to this issue, could you please create a new one to track it? The following information would be useful:

1. Is the report in Zendesk from the same pipeline where the error was seen?
2. If the user runs a new pipeline on the same branch/commit, does it consistently report the same error?

As part of the new issue, we may tweak the error message to explain the difference between schema validation and JSON parsing errors. I believe the error in this case is not related to schema validation, but we mention it in the error message because, in theory, if schema validation passes, then the JSON must be valid.

/cc @gitlab-org/secure/threat-insights-backend-team

@thiagocsf Thank you so much for the note, Thiago!

I also ran a pipeline with VALIDATE_SCHEMA: "true" and, as expected, no errors are displayed: https://gitlab.com/gitlab-org/secure/tests/test-for-issue-357038/-/pipelines/511947806.

That is interesting. The customer let me know that setting VALIDATE_SCHEMA: "true" did not resolve the errors. I will double-check on this and follow-up if I learn anything of note.

Since the error we're seeing is not related to this issue, could you please create a new one to track it? The following information would be useful:

1. Is the report in Zendesk from the same pipeline where the error was seen?
2. If the user runs a new pipeline on the same branch/commit, does it consistently report the same error?

Absolutely! I will work with the customer to collect this info and will follow back up on this thread with a link to that issue.

As part of the new issue, we may tweak the error message to explain the difference between schema validation and JSON parsing errors.

That sounds like a great idea.

wondering aloud Would it make more sense for me to create two issues?:

• One to tweak the error message to make the difference between schema validation and JSON parsing errors clearer
• One to better understand the specific JSON parsing errors that this customer is encountering

Thanks!

-- Brie

Would it make more sense for me to create two issues?

Good idea, @bcarranza. Thanks again!

Hi @thiagocsf! I am following up for Brie while she is OOO.

The customer in this ticket has 2 questions I was hoping you would be able to assist with.

• Are there security issues with our code that is not being identified due to this error?
• Is there something on our end that we need to do to correct the issue?

@abuerer

Are there security issues with our code that is not being identified due to this error?

Possibly. However, when I imported the report supplied by the customer, I didn't get any errors. This tells me that if they re-run the pipeline and don't get errors either, all findings should be ingested by GitLab and appear in the Vulnerability Report.

So if anyone ever gets this error, retries the same pipeline, and gets no errors, it's almost certain that they haven't missed anything (I say "almost" because there could be other external factors, but it's not worth worrying about these).

We'll be adjusting this error message to make it clear whether the problem was caused by an invalid report.

In this case, I believe that there was some other underlying issue (a DB timeout perhaps) that caused the error message. (FYI @minac)

Is there something on our end that we need to do to correct the issue?

Since there were no schema validation problems with the report that they gave me, there's nothing to follow-up with the analyzer teams at this stage.

If they continue to experience this, I'd like to engage to collect more information to help us diagnose the problem.

@thiagocsf Thank you!

I checked with the customer and they do continue to see the issue if they rerun the pipeline. Is there specific information you would like me to request from the customer?

@abuerer, yes please. Could they please confirm that the artifact in the Zendesk ticket generates the error? I tried it myself and didn't get any errors.

To be sure, we could ask them to upload all the artifacts from the latest pipeline that generated errors. This will rule-out any problems with the reports themselves.

If that doesn't bring anything up, we'll move-on to investigating their instance - it might be more efficient to do it on a screen-sharing session.

Would you mind creating a confidential issue for this customer so we can freely discuss their setup and exchange files?

Thank you!

@thiagocsf They have uploaded the artifacts to the ticket. Thank you!

@thiagocsf @abuerer Thank you both for your work on this. The customer followed up and let us know that the recent fix worked! I believe that the fix they are referring to is !87702 (merged). We are marking the ticket Solved! Thanks!

added customer label

@gonzoyumo, there's a disproportionate number of DS validation failures. I can't tell you yet what analyzer/scanner version is producing the reports just yet (waiting for !84522 (merged)), so I'm not sure if you can do anything about it.

Most failures are using schema version 14.0.0, so perhaps this is an older version of a DS analyzer?

From kibana

@thiagocsf that could indeed be any older version of any analyzer. Though that sounds odd to have such a distinct behavior for DS vs the other categories

We've identified a non-official analyzer that could be a source of these warnings but can't tell how it's used.

Is there a way to get the raw data and see if these comes from the same account/namespace?

EDIT: Just saw the more recent messages in the other thread :/

@gonzoyumo, the MR to add scanner id/version to kibana has been set to MWPS (!84522 (merged)). This should help us figure-out where it's coming from. I'll update when we have more data.

@gonzoyumo, scanner_id and version are now available in the logs and on the dashboard.

Thanks @thiagocsf, this is awesome data! It is definitely re-assuring as we can see the vast majority of failures come from Retire.JS and bundler audit, which is expected

We still see a few erros from the gemnasium analyzer and even on very recent version. It might be worth having a quick look at it.

The Deprecated schema usage reported for gemnasium are for versions that are a year old or more so I see no concern here.

cc @NicoleSchwartz

mentioned in issue #358739 (closed)

mentioned in merge request !84993 (merged)

mentioned in issue gitlab-org/secure/general#226 (closed)

mentioned in merge request gitlab-com/www-gitlab-com!102483 (merged)

Looking at the charts and logs today for the past 7 days, there are no new significant errors that need to be communicated to analyzer groups.

Update for today: still nothing of major concern in the charts.

@dappelt, IIRC you were involved with package hunter. It looks like it's using a very old version of the schema, and GitLab %15.0 will reject those reports. If I create an issue to handle this, do you know who should I assign it to?

@thiagocsf FYI, I updated the report format. It validates now against the schema. The latest pipeline doesn't show an error for the package hunter job anymore https://gitlab.com/gitlab-org/gitlab/-/pipelines/527551755/security

Excellent! Many thanks, @dappelt!

@gonzoyumo, FYI the yarn DS warnings are now errors since the enforce_security_report_validation FF is enabled on gitlab-org/gitlab. It will be enabled globally on production on the week of 16/May.

@gonzoyumo do i need to make a bug for this? also yarn-audit? that doesn't match the name pattern of any of our 3 scanners? cc @gitlab-org/secure/composition-analysis-be @sam.white

@thiagocsf @NicoleSchwartz the job is called yarn-audit in the CI config but the underlying analyzer is npm-audit. The same analyzer works for both npm and yarn: https://gitlab.com/gitlab-org/security-products/analyzers/npm-audit. This analyzer should have been fixed last week with gitlab-org/security-products/analyzers/npm-audit!18 (merged) but apparently we still have some errors. @julianthome would you be available to have another look?

@gonzoyumo I think we have to update the GitLab CI configuration. I just prepared a corresponding MR: !86459 (closed)

totally missed that... Good catch @julianthome!

@gonzoyumo @julianthome A large customer has reported similar error for one of their pipelines (Zendesk ticket - Internal).

As I understand, the fix for this should be available with the merge request !86459 (closed) which is set for 15.1, correct? Thanks in advance.

mhhh ... If I am interpreting the warning above correctly it looks as if the reports were generated by dast ... If this is the case than this issue won't be addressed by !86459 (closed) which only addresses the report generation for the npm-audit dependency scanning analyzer.

/cc @sethgitlab

mentioned in merge request !86459 (closed)

Closing this issue as %15.0 is now ready. Any failures from here on should be considered by the responsible analyzer teams.

closed

mentioned in issue gitlab-org/quality/triage-reports#7679 (closed)

mentioned in merge request !89967 (merged)