Skip to content

Introduce a model for npm metadata

🌱 Context

The metadata endpoints at the project and instance levels of npm Registry are the top 2 slowest endpoints for the Package Registry.

In the investigation on how to improve performance of those endpoints it was decided to generate a metadata using a background worker, store it in the object storage and to return it when the endpoints are hit instead of generating response "on the fly". Basically to have a cache.

Every active npm package will have a metadata cache.

At the moment npm Registry supports only abbreviated metadata format that will be stored in the JSON format in the object storage. Later, full metadata format will be added.

A metadata cache will be updated on the particular actions like a new version of package is uploaded or deleted, or a new tag added/updated, etc. All that actions will trigger an update of metadata cache through a background worker.

What does this MR do and why?

This MR adds a new model to operate and later to store npm Registry metadata cache, and a new table packages_npm_metadata_caches to the database

I excluded an uploader from this MR and will add it later together with the client that uses it.

💾 Database migrations

Migration up

$ rake db:migrate:main
main: == 20230313181536 CreatePackagesNpmMetadataCaches: migrating ==================
main: -- create_table(:packages_npm_metadata_caches)
main: -- quote_column_name(:file)
main:    -> 0.0000s
main:    -> 0.0052s
main: == 20230313181536 CreatePackagesNpmMetadataCaches: migrated (0.0680s) =========

main: == 20230322145403 AddProjectIdForeignKeyToPackagesNpmMetadataCaches: migrating 
main: -- transaction_open?()
main:    -> 0.0001s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE packages_npm_metadata_caches\nADD CONSTRAINT fk_ada23b1d30\nFOREIGN KEY (project_id)\nREFERENCES projects (id)\n\nON DELETE SET NULL\nNOT VALID;\n")
main:    -> 0.0016s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- execute("ALTER TABLE packages_npm_metadata_caches VALIDATE CONSTRAINT fk_ada23b1d30;")
main:    -> 0.0024s
main: -- execute("RESET statement_timeout")
main:    -> 0.0002s
main: == 20230322145403 AddProjectIdForeignKeyToPackagesNpmMetadataCaches: migrated (0.0223s)

Migration down

$ rake db:rollback:main STEP=2
main: == 20230322145403 AddProjectIdForeignKeyToPackagesNpmMetadataCaches: reverting 
main: -- transaction_open?()
main:    -> 0.0000s
main: -- remove_foreign_key(:packages_npm_metadata_caches, {:column=>:project_id})
main:    -> 0.0799s
main: == 20230322145403 AddProjectIdForeignKeyToPackagesNpmMetadataCaches: reverted (0.0970s) 

main: == 20230313181536 CreatePackagesNpmMetadataCaches: reverting ==================
main: -- drop_table(:packages_npm_metadata_caches)
main:    -> 0.0010s
main: == 20230313181536 CreatePackagesNpmMetadataCaches: reverted (0.0036s) =========

🔬 How to set up and validate locally

  1. In rails console create a new instance of Packages::Npm::MetadataCache model

    Packages::Npm::MetadataCache.create(project_id: Project.first.id, package_name: '@root/test')
  2. Check that a new instance of Packages::Npm::MetadataCache is successfully created.

🏁 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #393633 (closed)

Edited by Dzmitry Meshcharakou

Merge request reports