Skip to content

Add service to index nuget symbol files

Moaz Khalifa requested to merge 416177-index-nuget-symbol-files into master

What does this MR do and why?

In !129916 (merged), we created the packages_nuget_symbols database table. In this MR, we introduce the services needed to create records to be inserted in this table.

In the NuGet Repository, we support publishing symbol packages. But to make use of those packages, they need to be indexed.

The indexing process means extracting the debugging files .pdb from the uploaded symbol package .snupkg, and for each extracted file .pdb we create a corresponding database record in the packages_nuget_symbols table.

The packages_nuget_symbols database record holds the following information about the .pdb file:

  • file: the reference to the .pdb file on the object store.
  • signature: This is a unique GUID and it's embedded in the .pdb file. It's extracted from the file by Packages::Nuget::ExtractSymbolSignatureService. When debuggers such as Visual Studio need to retrieve the symbol debugging files from symbol servers, they send this signature to the server to retrieve the correct files.
  • file_path: It's the path of the .pdb file in the directory structure of the symbol package .snupkg. For ex: "lib/netstandard2.0/MyPackage.pdb"
  • size: the size of the .pdb file.
  • package_id: the id of the parent NuGet package in the packages_packgaes table.

Now to summarize the flow of pushing a symbol package to the NuGet Repository:

  1. The user would push a symbol package using nuget.cli:
nuget push package.1.0.0.snupkg -Source gitlab
  1. When the server receives the package file package.1.0.0.snupkg, it processes it as follows:
    1. Extract the metadata .nuspec file which contains the necessary metadata such as the name and the version of the package.
    2. Extract the .pdb files and index them; meaning create a database record for each file in the packages_nuget_symbols table, and upload the file to the object storage. (this step is introduced in this MR)

Technical implementation:

Before this MR, we had one service responsible for fetching the package file from the object store and extract the metadata .nuspec file from it. But in this MR, I needed to separate this service into two services: one is responsible for the package file from the object store, and the other is responsible for extracting the metadata .nuspec file.

This refactor was needed because the newly introduced service Packages::Nuget::CreateSymbolFilesService needs the fetched package file to extract the .pdb files from it. That means we need to fetch the package file once, and then use it twice: in the metadata .nuspec file extraction service, and in the symbol .pdb files creation service.

Now we can summarize how the services are structured:

  • Packages::Nuget::ProcessPackageFileService: New service which is responsible for downloading the package file from the object store and then passing it to the other two services;
  • Packages::Nuget::ExtractMetadataFileService: Existing refactored service that receives the downloaded package zip file and extracts the .nuspec metadata file.
  • Packages::Nuget::CreateSymbolFilesService: New service that receives the downloaded package zip file and indexes the symbol .pdb files.

I wanted the symbols creation service to be as quiet as possible. It doesn't complain or raise exceptions. It just tries to create database records and upload files for the extracted .pdb files. If there's a uniqueness violation or unsuccessful signature extraction, the new record will silently fail and be skipped. I thought this could be beneficial in the beginning since we don't want to interrupt the main package uploading process with any raised exceptions from the newly introduced indexing service. Once it is deemed stable and functional, we can enhance the flow as needed.

The indexing service is behind the index_nuget_symbol_files feature flag to gradually roll out the feature and revert easily in case of any errors.


Implementation of Packages::Nuget::ExtractSymbolSignatureService:

To implement the logic of extracting the signature from the .pdb files, I used this page as a reference to know about the structure of the signature and how it's constructed. However, the implementation of the service is done after several trial and error attempts. Therefore, it's not a well-known implementation because of the lack of such a thing. I tried to look for a ready-to-use solution on how to extract this piece of information from .pdb files with no luck.


Known limitation:

The implementation targets only the Microsoft .Net portable PDB format files. Other symbol files will not be indexed correctly.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Make sure you have the NuGet CLI installed (see nuget docs for links to installation pages).
  2. Add a project as your NuGet source in your local GitLab NuGet Repository:
   nuget source Add -Name localhost -Source "http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json" -UserName <gitlab_username> -Password <personal_access_token>
  1. In the rails console, enable the feature flag for the project:
Feature.enable(:index_nuget_symbol_files, Project.find(<project_id>))
  1. Push a NuGet package that has a symbol to your project. You can test using this one. Download both the .nupkg & .snupkg files in the same directory, and then push the .nupkg. The .snupkg file should be pushed automatically after the .nupkg is pushed.
  nuget push Package.nupkg -Source localhost
  1. Once the package is published, in the rails console verify that the symbol records have been created:
Packages::Nuget::Symbol.where(package_id: <package_id>)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #416177 (closed)

Edited by Moaz Khalifa

Merge request reports