Skip to content

Update package max file sizes to sensible defaults

Steve Abrams requested to merge 240949-update-package-limit-defaults into master

What does this MR do? 🔍

Maximum file size limits were very recently (2020-08-24) added to each package type in the GitLab package registry in the plan_limits table. The default for each limit was set to 50MB. This was a mistake and led to an ~S1 production incident due to the fact that Maven packages regularly exceed 50MB (sometimes reaching multiple gigabytes). 😞

For GitLab.com, we were able to simply update the limits for each plan to a higher limit, however, when 13.4 is released, the original 50MB defaults will be the defaults added to all self-managed instances. We do not want that to happen.

This MR updates the defaults to much higher sizes based on what we see on GitLab.com:

  • Conan: 3GB
  • Maven: 3GB
  • NPM: 500MB
  • NuGet: 500MB
  • PyPI: 3GB

And then sets the values to those same limits. The values need to be set because changing the default does not change the value that will be inserted when the columns are created in the previous migration that has a default set to 50MB.

Note: This will not have an effect on GitLab.com values as the plans already have values. It will only have an effect if a new plan is created (as in gold/silver/free). Then the new plan will receive these defaults.

Double Note: These limits are adjustable on self-managed instances. So although we are adding higher limits. Once released, admin users can then adjust as they see fit. The main concern of this MR is to prevent 400 errors and broken pipelines due to failed package uploads for self-managed instances.

Database 🐘

Update the defaults

Up migration ⬆

== 20200825154237 UpdatePackageFileSizePlanLimitsDefaults: migrating ==========
-- change_column_default(:plan_limits, :maven_max_file_size, {:from=>52428800, :to=>3221225472})
   -> 0.0044s
-- change_column_default(:plan_limits, :conan_max_file_size, {:from=>52428800, :to=>3221225472})
   -> 0.0024s
-- change_column_default(:plan_limits, :nuget_max_file_size, {:from=>52428800, :to=>524288000})
   -> 0.0023s
-- change_column_default(:plan_limits, :npm_max_file_size, {:from=>52428800, :to=>524288000})
   -> 0.0022s
-- change_column_default(:plan_limits, :pypi_max_file_size, {:from=>52428800, :to=>3221225472})
   -> 0.0020s
== 20200825154237 UpdatePackageFileSizePlanLimitsDefaults: migrated (0.0135s) =

Down migration ⬇

== 20200825154237 UpdatePackageFileSizePlanLimitsDefaults: reverting ==========
-- change_column_default(:plan_limits, :pypi_max_file_size, {:from=>3221225472, :to=>52428800})
   -> 0.0124s
-- change_column_default(:plan_limits, :npm_max_file_size, {:from=>524288000, :to=>52428800})
   -> 0.0031s
-- change_column_default(:plan_limits, :nuget_max_file_size, {:from=>524288000, :to=>52428800})
   -> 0.0029s
-- change_column_default(:plan_limits, :conan_max_file_size, {:from=>3221225472, :to=>52428800})
   -> 0.0027s
-- change_column_default(:plan_limits, :maven_max_file_size, {:from=>3221225472, :to=>52428800})
   -> 0.0028s
== 20200825154237 UpdatePackageFileSizePlanLimitsDefaults: reverted (0.0261s) =

Update the values

Up migration ⬆

== 20200902135542 UpdatePackageMaxFileSizePlanLimits: migrating ===============
-- quote_column_name("conan_max_file_size")
   -> 0.0000s
-- quote(3221225472)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"conan_max_file_size\" = '3221225472';\n")
   -> 0.0020s
-- quote_column_name("maven_max_file_size")
   -> 0.0000s
-- quote(3221225472)
   -> 0.0001s
-- execute("UPDATE plan_limits\nSET \"maven_max_file_size\" = '3221225472';\n")
   -> 0.0010s
-- quote_column_name("npm_max_file_size")
   -> 0.0000s
-- quote(524288000)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"npm_max_file_size\" = '524288000';\n")
   -> 0.0005s
-- quote_column_name("nuget_max_file_size")
   -> 0.0000s
-- quote(524288000)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"nuget_max_file_size\" = '524288000';\n")
   -> 0.0004s
-- quote_column_name("pypi_max_file_size")
   -> 0.0000s
-- quote(3221225472)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"pypi_max_file_size\" = '3221225472';\n")
   -> 0.0005s
== 20200902135542 UpdatePackageMaxFileSizePlanLimits: migrated (0.0052s) ======

Down migration ⬇

== 20200902135542 UpdatePackageMaxFileSizePlanLimits: reverting ===============
-- quote_column_name("conan_max_file_size")
   -> 0.0000s
-- quote(52428800)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"conan_max_file_size\" = '52428800';\n")
   -> 0.0015s
-- quote_column_name("maven_max_file_size")
   -> 0.0000s
-- quote(52428800)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"maven_max_file_size\" = '52428800';\n")
   -> 0.0009s
-- quote_column_name("npm_max_file_size")
   -> 0.0000s
-- quote(52428800)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"npm_max_file_size\" = '52428800';\n")
   -> 0.0005s
-- quote_column_name("nuget_max_file_size")
   -> 0.0000s
-- quote(52428800)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"nuget_max_file_size\" = '52428800';\n")
   -> 0.0005s
-- quote_column_name("pypi_max_file_size")
   -> 0.0000s
-- quote(52428800)
   -> 0.0000s
-- execute("UPDATE plan_limits\nSET \"pypi_max_file_size\" = '52428800';\n")
   -> 0.0005s
== 20200902135542 UpdatePackageMaxFileSizePlanLimits: reverted (0.0045s) ======

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

Related #240949 (closed)

Edited by Steve Abrams

Merge request reports