Hao Hu authored 3 months ago and Jamie Tanna committed 3 months ago

When deriving the underlying `package_name` for the Renovate datasource,
we've been using `depName` from the underlying package data exports.

For most cases, this gives us exactly what we want, but in some cases,
this presents a different a different `package_name`, as the `depName`
is generally intended to be for the "pretty" name for a package.

For instance:

    {
      "depType": "plugin",
      "depName": "org.sonarqube",
      "packageName": "org.sonarqube:org.sonarqube.gradle.plugin",
      "currentValue": "3.3",
      "datasource": "maven",
      "packageScope": "org.sonarqube",
      "currentVersion": "3.3",
      "isSingleVersion": true,
      "fixedVersion": "3.3"
    }

To correct this, we can make sure we now prioritise `packageName`.

This requires a bump of `compatible_since`, as this is a change to the
way that the data is derived, and may incur breakages to existing users'
queries.

When skipping conflict resolution - in the case there was only one
possible diff to select, we were missing out on valid data, resulting in
incomplete diffs.

In the case that we want to marshal the empty struct, we hit a
long-standing Go issue where an uninitialised slice is `nil` which
produces JSON `null` instead of `[]`.

As noted in #618, it would be convenient to be able to have a
machine-parseable format for our diff result.

This wires in the `--output json` flag, and allows exporting the data
accordingly, which requires:

- wiring in the flag
- tweaking our models to include JSON struct tags
- specifying `omitEmpty` on `Diffs`' children, as they should only
  appear when non-`null` or non-`nullable`.
- make sure that `DepTypes` is always initialised as an empty array, so
  we get a `[]` marshaled, not a `null`
- add a JSON Schema
  - with relevant examples
  - with `description`s via the database schema

This has only been validated via an online tool[0], despite
recommendations otherwise[1], but found OK due to this being all public
data.

Closes #618.

[0]: https://www.jsonschemavalidator.net/
[1]: https://www.jvt.me/posts/2020/09/01/against-online-tooling/

As a way to provide a more useful way to understand what's changed
between exports of Renovate data - via `renovate-graph` or our other
supported formats, we should introduce a separate cli,
`renovate-packagedata-diff`.

This could be used to i.e.:

- see what changes are coming in as part of a PR/MR
- summarise changes in `git diff` outputs for projects that source-track
  their `renovate-graph` exports
- wire into CI to i.e. raise warnings when changing important
  dependencies

It makes sense to be part of dependency-management-data, rather than
`renovate-graph`, as we already have some good understanding of the
structure here, and can provide a more convenient output format.

Additionally, producing a Go binary will be more convenient i.e. for CI
builds.

This should - at least medium term - also produce a stable JSON response
output, which can be used for scripting.

For now, we will only provide a human-readable summary.

At some point, we may also expose this as part of a Go API.

To do this, we need to:

- create a new CLI
  - which can provide `--debug` logging
- present the output to the user, in a tabular, colourised fashion
- handle being called ad-hoc or as a `GIT_EXTERNAL_DIFF` tool
- create a new `CompareFiles` function, which orchestrates the overall
  diffing between two package dumps
- perform the difficult job of determining which modifications are
  actually modifications, and not presenting as addition/deletion
- wire in the command to `gendoc`

Note that there is a fair bit of inline code commentary, and areas which
look a little "spooky" but don't have comments could be that in the time
it's taken to finish implementation have since been forgotten, or test
cases lost.

There are test cases in the -example project, under
`/renovate-packagedata-diff`.

Closes #616.

As we now track some `Metadata` for the parsed Renovate data, we can
also track what the underlying source of data is, for other purposes.

We can use a `map` to create a Set of `ParsingSource`s, to ensure that
it's a valid type.

We also need to wire this in now, as it's used across all parsing types.

As part of #616, we want to flag differences in major version of
Renovate.

To do so, we should parse this out of the Renovate data, where available.

This is a step towards #36.

Instead of using an unnamed struct, which can be awkward to instantiate,
we should instead use a named type.

As it's going to be useful as part of future commits.

Thanks to Yannic on the Gopher Slack for suggesting using the
`spancheck` linter.

There are a few places where we weren't correctly calling `span.End()`,
or that `spancheck` thought we weren't calling it, and needs to be
explicitly told to ignore it.

As it makes more sense for what it is.

To simplify the configuration of the OTel exporters, we can utilise
`autoexport` which performs the environment variable switches that we
were previously doing, and handles a few other options.

This slightly changes our configuration for metrics, which hopefully
doesn't make an impact - or won't when we start producing metrics.

This also requires a bump to the `go` directive, as the package requires
a minimum of 1.22.7.

To make sure that we don't hamper experience for our users too much, we
should add an explicit 1s timeout (down from default 5s).

To make sure that we always set up the Span, in an automagically managed
process that always occurs before any processing, we should make the
`StartSpanForCommand` call execute in the `PersistentPreRunE` and
`PersistentPostRunE`.

This also makes `StartSpanForCommand` update the `context.Context` in
the function itself, so we're always using the correct child
`context.Context`.

As part of #621.

As part of #621.

As part of #621.

As part of #621.

As part of #621.

As part of #621.

As part of #621.

As part of a prefactor to allow for further instrumentation in #621, we
should pass a `context.Context` to our parsing code.

As part of #621, we can add some custom instrumentation to the
generation of advisories for AWS infrastructure.

This could be split into separate commits, but is not worthwhile.

As part of a prefactor to allow for further instrumentation in #621, we
should pass a `context.Context` to our parsing code.

This could be split into separate commits, but is not worthwhile.

As part of #621, we can add some custom instrumentation to the
generation of Policy violations, or evaluation of Policies.

As part of #621, we can add some custom instrumentation to the
generation of "missing data", via deps.dev.

As part of #621, we can add some custom instrumentation to the
generation of Libyear metadata.

As part of #621, we want to instrument `dmd`.

This prefactor makes sure that the Security Scorecards client handles
the `context.Context` propagation, so we can capture calls to
`securityscorecards.dev` in the same span as other actions, as well as
when importing.

As part of #621, we can add some custom instrumentation to the
generation of advisories from Dependency Health.

As part of #621, we can add some custom instrumentation to the
generation of advisories for AWS infrastructure.

This could be split into separate commits, but is not worthwhile.

As part of #621, we can add some custom instrumentation to the
generation of advisories from the deps.dev, endoflife.date, etc tables.

As part of #621, we can add some custom instrumentation to the
generation of advisories for the "inbuilt" advisories.

As part of #621, we can add some custom instrumentation to the
generation of advisories for `-contrib`.

As part of #621, we can add some custom instrumentation to the
generation of advisories for `-contrib`.

As part of #621, we can add some custom instrumentation to our calls out
to deps.dev.

As part of #621, we want to instrument `dmd`.

This prefactor makes sure that the deps.dev client handles the
`context.Context` propagation, so we can capture calls to deps.dev in
the same span as other actions.

As part of #621, we want to have insight across everything `dmd` does.

To automagically instrument our HTTP calls, we can use `otelhttp`'s
inbuilt automagic `http.Transport` for this.

This also improves the duplication by calling out to a central method
for creating our HTTP client.

As part of #621, we can add some custom instrumentation to better
understand how `import renovate` operates.