Skip to content

Re-orchestrate updating NuGet package flow from downloaded package file

Moaz Khalifa requested to merge 426207-reorchestrate-updating-nuget-package-from-downloaded-file into master

What does this MR do and why?

In Add service to index nuget symbol files (!131567 - merged), a new service was added to create symbol files .pdb records in the database. Those .pdb files are embedded in the uploaded .snupkg files. Thus, we need to download the .snupkg file from the Object Storage first and then extract the .pdb files from it.

On the other hand, the downloaded .snupkg file is used to extract the metadata file nuspec from it. That means we should download the .snupkg only once and then use it to do two things:

  1. extract the metadata file nuspec
  2. extract the symbol files .pdb

In order to do so, the ProcessPackageFileService was introduced in !131567 (merged). It was responsible for downloading the package file and then passing it to the metadata extraction service & the symbol creation service.

So this was the flow of the services:

flowchart BT
    Packages::Nuget::ExtractionWorker --> Packages::Nuget::UpdatePackageFromMetadataService 
    Packages::Nuget::UpdatePackageFromMetadataService --> Packages::Nuget::MetadataExtractionService 
    Packages::Nuget::MetadataExtractionService --> Packages::Nuget::ProcessPackageFileService
    Packages::Nuget::ProcessPackageFileService --> Packages::Nuget::ExtractMetadataFileService
    Packages::Nuget::ProcessPackageFileService --> Packages::Nuget::Symbols::CreateSymbolFilesService

The problem with this flow is that we need to use the downloaded file from Packages::Nuget::ProcessPackageFileService inside Packages::Nuget::UpdatePackageFromMetadataService & Packages::Nuget::MetadataExtractionService. This is against the direction of the flow and made us unable to present the orchestration between the services in a correct way.

Also, we faced a problem as we were unable to pass the correct package to the Packages::Nuget::Symbols::CreateSymbolFilesService when creating the symbol files. That's because we are detecting the package in Packages::Nuget::UpdatePackageFromMetadataService while we are creating the symbol files after Packages::Nuget::ProcessPackageFileService downloaded the package file.

To summarize, we needed to rearrange the flow and start with downloading the file and then passing it to whatever services would use it. The updated flow is as follows:

flowchart BT
    Packages::Nuget::ExtractionWorker --> Packages::Nuget::ProcessPackageFileService
    Packages::Nuget::ProcessPackageFileService --> Packages::Nuget::UpdatePackageFromMetadataService
    Packages::Nuget::UpdatePackageFromMetadataService --> Packages::Nuget::MetadataExtractionService
    Packages::Nuget::UpdatePackageFromMetadataService --> Packages::Nuget::Symbols::CreateSymbolFilesService
    Packages::Nuget::MetadataExtractionService --> Packages::Nuget::ExtractMetadataFileService

By starting the flow from downloading the file, we can make sure that we are able to extract whatever file we need and maintain a predictable and one-directional orchestration between the services.

Screenshots or screen recordings

Not applicable.

How to set up and validate locally

Follow the same steps mentioned here. This is a refactoring MR so uploading a nuget package should work fine as before.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #426207 (closed)

Edited by Moaz Khalifa

Merge request reports