Skip to content

Add file_metadata jsonb

Aditya Tiwari requested to merge 341318-add-more-info-to-corpus into master

What does this MR do and why?

  1. This MR adds file_metadata field of type jsonb to store extra information related to the associated package file needed by frontend.
  2. We are populating the file_metadata field asynchronously using CorpusFileMetadataUpdateWorker. This worker will be triggered in after_create callback.
  3. CorpusFileMetadataUpdateWorker parses the zip file and fetch the required metadata from the zip. Currently, we are only supporting the zip file.

Steps to test

  1. Create a package_file using the following command. The command will also create a package.
curl --header "PRIVATE-TOKEN: uzsLZ6teL2oWJobTmTJE" \
              --upload-file ~/Downloads/zip_2MB.zip \
              "http://gitlab.localdev:3000/api/v4/projects/1/packages/generic/corpus8_package/0.0.1/zip_2MB.zip?status=hidden&select=package_file"

Note: a. Please adjust the PRIVATE-TOKEN and project_id as per your setup. b. You can use attachedzip_2MB.zip zip file or create your own.

  1. The above API will respond with the package_id. Use the package_id to create a corpus.
AppSec::Fuzzing::Coverage::Corpus.create(project_id: 1, package_id: 33)
  1. The create will respond with empty file_metadata and trigger a background job.
file_metadata: {}
  1. Reload the object and it should show the file_metadata as follows:
file_metadata: {"total_size"=>2036861, "total_files"=>3}

Note: Since this MR is introducing a new background job you may have to restart your sidekiq with command gdk restart rails-background-jobs

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #341318 (closed)

Edited by Aditya Tiwari

Merge request reports