Slowness rendering JSON data for PipelinesController#index.json

I've been noticing that it takes several seconds to load https://gitlab.com/gitlab-org/gitlab/pipelines.json even though SQL/Gitaly queries seem fairly fast:

image

Attached is the flamegraph from stackprof:

flamegraph.zip

image

$ stackprof pipeline.sample
==================================
  Mode: wall(1000)
  Samples: 6559 (1.03% miss rate)
  GC: 285 (4.35%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
      2969  (45.3%)        2926  (44.6%)     ActionDispatch::Journey::Formatter#missing_keys
       285   (4.3%)         285   (4.3%)     (garbage collection)
       179   (2.7%)         179   (2.7%)     ActiveRecord::Associations#association_instance_get
       140   (2.1%)         130   (2.0%)     ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#exec_no_cache
       122   (1.9%)         122   (1.9%)     ActiveModel::Attribute#initialize
       250   (3.8%)         121   (1.8%)     ActiveModel::LazyAttributeHash#[]
       164   (2.5%)          98   (1.5%)     ActionDispatch::Journey::Format#evaluate
       305   (4.7%)          93   (1.4%)     ActionDispatch::Journey::Formatter#extract_parameterized_parts
      4001  (61.0%)          73   (1.1%)     block (2 levels) in define_url_helper
      3595  (54.8%)          71   (1.1%)     ActionDispatch::Journey::Formatter#generate
        71   (1.1%)          71   (1.1%)     ActionDispatch::Routing::RouteSet::NamedRouteCollection::UrlHelper#handle_positional_args
        65   (1.0%)          65   (1.0%)     ActionDispatch::Journey::Router::Utils::UriEncoder#escape
        83   (1.3%)          62   (0.9%)     #<Module:0x00007f36cb3a52d8>.clean_backtrace
        61   (0.9%)          61   (0.9%)     ActiveRecord::Result#hash_rows
        58   (0.9%)          58   (0.9%)     BatchLoader#__replace_with!
        88   (1.3%)          55   (0.8%)     #<Module:0x00007f36e32444f8>.path_for
        52   (0.8%)          52   (0.8%)     ActiveSupport::ToJsonWithActiveSupportEncoder#to_json
        51   (0.8%)          51   (0.8%)     ActiveSupport::NumericWithFormat#to_s
        48   (0.7%)          48   (0.7%)     Delegator.const_missing
      3801  (58.0%)          48   (0.7%)     ActionDispatch::Routing::RouteSet#url_for
        82   (1.3%)          44   (0.7%)     ActiveModel::Type::Helpers::TimeValue#new_time
        57   (0.9%)          44   (0.7%)     block (2 levels) in class_attribute
        95   (1.4%)          43   (0.7%)     ActiveSupport::JSON::Encoding::JSONGemEncoder::EscapedString#to_json
        55   (0.8%)          43   (0.7%)     ActiveModel::AttributeMethods#respond_to?
        42   (0.6%)          42   (0.6%)     ActiveRecord::Reflection::AssociationReflection#foreign_key
        39   (0.6%)          39   (0.6%)     Grape::Entity::Exposure::Base#attr_path
        40   (0.6%)          39   (0.6%)     ActiveSupport::JSON::Encoding::JSONGemEncoder#jsonify
        38   (0.6%)          38   (0.6%)     ActiveRecord::Transactions#update_attributes_from_transaction_state
        38   (0.6%)          38   (0.6%)     ActiveRecord::Base.default_timezone
        37   (0.6%)          37   (0.6%)     ActionDispatch::Routing::RouteSet::Generator#normalize_options!
$ stackprof pipeline.sample --method=ActionDispatch::Journey::Formatter#missing_keys
ActionDispatch::Journey::Formatter#missing_keys (/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/formatter.rb:139)
  samples:  2926 self (44.6%)  /   2969 total (45.3%)
  callers:
    2969  (  100.0%)  ActionDispatch::Journey::Formatter#generate
    2951  (   99.4%)  ActionDispatch::Journey::Formatter#missing_keys
  callees (43 total):
    2951  ( 6862.8%)  ActionDispatch::Journey::Formatter#missing_keys
      27  (   62.8%)  #<Module:0x00007f36e2f696e8>.===
      16  (   37.2%)  ActionDispatch::Journey::Route#required_parts