Persist NuGet package dependencies in metadata table

Moaz Khalifarequested to merge
440403-package-registry-fails-to-resolve-nuget-dependencies-correctly into master

Context

GitLab's NuGet Repository doesn't persist the target frameworks that don't have any dependencies.

For example, if a package was published with the following metadata:

XML 
<dependencies>
    <group targetFramework=".NETStandard1.3">
        <dependency id="System.Net.Http" version="4.3.1" />
        <dependency id="System.Xml.XmlDocument" version="4.3.0" />
        <dependency id="System.Xml.XPath" version="4.3.0" />
        <dependency id="System.Xml.XPath.XmlDocument" version="4.3.0" />
    </group>
    <group targetFramework=".NETFramework4.0" />
    <group targetFramework=".NETFramework4.0-Client" />
    <group targetFramework=".NETFramework4.5" />
    <group targetFramework=".NETCore4.5" />
    <group targetFramework=".NETStandard1.6">
        <dependency id="System.Net.Http" version="4.3.1" />
        <dependency id="System.Xml.XmlDocument" version="4.3.0" />
        <dependency id="System.Xml.XPath" version="4.3.0" />
        <dependency id="System.Xml.XPath.XmlDocument" version="4.3.0" />
        <dependency id="NETStandard.Library" version="1.6.1" />
    </group>
    <group targetFramework=".NETPortable0.0-net45+netcore45+wp8+MonoAndroid+MonoTouch" />
    <group targetFramework=".NETPortable0.0-net45+netcore45+wpa81+wp8+MonoAndroid+MonoTouch" />
    <group targetFramework="Silverlight5.0" />
    <group targetFramework="UAP10.0">
        <dependency id="Microsoft.NETCore.UniversalWindowsPlatform" version="5.2.3" />
        <dependency id="System.Net.Http" version="4.3.2" />
        <dependency id="System.Xml.XPath" version="4.3.0" />
        <dependency id="System.Xml.XPath.XDocument" version="4.3.0" />
        <dependency id="System.Xml.XPath.XmlDocument" version="4.3.0" />
    </group>
    <group targetFramework=".NETStandard2.0">
        <dependency id="System.Net.Http" version="4.3.2" />
        <dependency id="System.Xml.XmlDocument" version="4.3.0" />
        <dependency id="System.Xml.XPath.XmlDocument" version="4.3.0" />
        <dependency id="System.Xml.XPath" version="4.3.0" />
    </group>
</dependencies>

we can see some target frameworks that have no dependencies:

  • .NETFramework4.0
  • .NETFramework4.0-Client
  • .NETFramework4.5
  • .NETCore4.5
  • .NETPortable0.0-net45+netcore45+wp8+MonoAndroid+MonoTouch
  • .NETPortable0.0-net45+netcore45+wpa81+wp8+MonoAndroid+MonoTouch
  • Silverlight5.0

Those targets are ignored by GitLab's NuGet Repository, and this leads to the dependency confusion described in Package registry fails to resolve NuGet depende... (#440403).

What does this MR do and why?

The current dependency persistence design isn't ready to host those empty targets, which means it only support persisting dependencies that have name and version, while in the case of empty targets, we don't have any dependencies.

The solution I came up with is to store the whole dependencies for each package in a new JSONB dependencies field in the packages_nuget_metadata table. This way, we can persist empty targets along with the non-empty ones in one object, and then return this object when the metadata endpoint is requested.

The fix is behind a feature flag. When the FF is enabled, we persist dependencies in the new filed on the package publication, but we still also perform the legacy path of persisting dependencies. So, now we would have duplicate data, but I think this is fine since the end goal will be to replace the legacy path with the new one.

Once we globally roll out the fix, and it deemed stable, we can start backfilling the existing data to the new dependencies column and discard the legacy path.

References

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. Enable the feature flag new_nuget_registry_dependencies

  2. Publish this package to the NuGet Repository:

    dotnet nuget push htmlagilitypack.1.6.5.nupkg --source http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json --api-key <PAT>

  3. Disable the feature flag new_nuget_registry_dependencies

  4. Compare the dependencies we return with nuget.org

    # from gitlab registry
curl -s -H "Private-Token: <PAT>" "http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/metadata/HtmlAgilityPack/1.6.5.json" | jq '.catalogEntry.dependencyGroups'

# from nuget.org
curl "https://api.nuget.org/v3/catalog0/data/2018.10.15.07.01.06/htmlagilitypack.1.6.5.json" | jq '.dependencyGroups'

    Notice how the response is different: mainly all target frameworks that have no dependencies are missing from GitLab's response, while present in nuget.org's response.

  5. Re-enable the feature flag new_nuget_registry_dependencies

  6. Fetch the dependencies again and make sure that our response is similar to what we get from nuget.org.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #440403

Edited by Moaz Khalifa

Merge request reports