Skip to content

Add custom pluralization to FastGettext

Peter Leitzen requested to merge pl-gettext-pluralization into master

What does this MR do and why?

This speeds up pluralization when using n_(...) by 17x.

It also side-steps any potential security concerns related to eval in FastGettext.

Example failure message

Each locales must have a pluralization mapping otherwise the specs will fail with the following error:

  2) Gitlab::I18n::Pluralization.call with available locales locale: "de" supports pluralization
     Failure/Error:
               raise ArgumentError, <<~MESSAGE
                 Missing pluralization rule for locale #{locale.inspect}.

                 #{suggestion}
               MESSAGE

     ArgumentError:
       Missing pluralization rule for locale "de".

       Add the following line to /home/peter/devel/gitlab/gdk/gitlab/lib/gitlab/i18n/pluralization.rb:

         MAP = {
           ...
           "de" => ->(n) { (n != 1) },
           ...
         }.freeze

       This rule was extracted from /home/peter/devel/gitlab/gdk/gitlab/locale/de/gitlab.po.
     # ./lib/gitlab/i18n/pluralization.rb:53:in `block in <module:Pluralization>'
     # ./lib/gitlab/i18n/pluralization.rb:63:in `fetch'
     # ./lib/gitlab/i18n/pluralization.rb:63:in `call'
     # ./spec/lib/gitlab/i18n/pluralization_spec.rb:9:in `block (3 levels) in <top (required)>'
     # ./spec/lib/gitlab/i18n/pluralization_spec.rb:22:in `block (5 levels) in <top (required)>'
     # ./lib/gitlab/i18n.rb:107:in `with_locale'
     # ./spec/lib/gitlab/i18n/pluralization_spec.rb:13:in `block (4 levels) in <top (required)>'
     # ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'

Benchmark

Comparison:
         plural fast:   717904.2 i/s
         plural slow:    40457.3 i/s - 17.74x  slower
Code + Results
# frozen_string_literal: true

require "benchmark/ips"

Gitlab::I18n::Pluralization.install! if ENV['FAST']

p FastGettext.pluralisation_rule

FastGettext.locale = "pl_PL"

p n_("%d day", "%d days", 0)
p n_("%d day", "%d days", 1)
p n_("%d day", "%d days", 5)

Benchmark.ips do |x|
  name = "plural #{ENV['FAST'] ? 'fast' : 'slow'}"

  x.report name do
    n_("day", "days", 1)
  end

  x.compare!
  x.save! 'fast_gettext'
end
$ bin/rails runner foo.rb
WARNING: This version of GitLab depends on gitlab-shell 14.17.0, but you're running 14.15.0. Please update gitlab-shell.
#<Proc:0x0000565246e3f6e0 /home/peter/.dotfiles/asdf/installs/ruby/3.0.5/lib/ruby/gems/3.0.0/gems/fast_gettext-2.1.0/lib/fast_gettext/storage.rb:17 (lambda)>
"%d dni"
"%d dzień"
"%d dni"
Warming up --------------------------------------
         plural slow     4.125k i/100ms
Calculating -------------------------------------
         plural slow     40.457k (±15.2%) i/s -    193.875k in   5.045078s

Comparison:
         plural fast:   512635.9 i/s
         plural slow:    40457.3 i/s - 12.67x  slower

peter@happy ~/devel/gitlab/gdk/gitlab on (pl-gettext-pluralization ?M)
$ FAST=1 bin/rails runner foo.rb
WARNING: This version of GitLab depends on gitlab-shell 14.17.0, but you're running 14.15.0. Please update gitlab-shell.
Gitlab::I18n::Pluralization
"%d dni"
"%d dzień"
"%d dni"
Warming up --------------------------------------
         plural fast    65.010k i/100ms
Calculating -------------------------------------
         plural fast    717.904k (±11.2%) i/s -      3.576M in   5.040367s

Comparison:
         plural fast:   717904.2 i/s
         plural slow:    40457.3 i/s - 17.74x  slower

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Peter Leitzen

Merge request reports