Slowness rendering JSON data for PipelinesController#index.json
I've been noticing that it takes several seconds to load https://gitlab.com/gitlab-org/gitlab/pipelines.json even though SQL/Gitaly queries seem fairly fast:
Attached is the flamegraph from stackprof:
$ stackprof pipeline.sample
==================================
Mode: wall(1000)
Samples: 6559 (1.03% miss rate)
GC: 285 (4.35%)
==================================
TOTAL (pct) SAMPLES (pct) FRAME
2969 (45.3%) 2926 (44.6%) ActionDispatch::Journey::Formatter#missing_keys
285 (4.3%) 285 (4.3%) (garbage collection)
179 (2.7%) 179 (2.7%) ActiveRecord::Associations#association_instance_get
140 (2.1%) 130 (2.0%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#exec_no_cache
122 (1.9%) 122 (1.9%) ActiveModel::Attribute#initialize
250 (3.8%) 121 (1.8%) ActiveModel::LazyAttributeHash#[]
164 (2.5%) 98 (1.5%) ActionDispatch::Journey::Format#evaluate
305 (4.7%) 93 (1.4%) ActionDispatch::Journey::Formatter#extract_parameterized_parts
4001 (61.0%) 73 (1.1%) block (2 levels) in define_url_helper
3595 (54.8%) 71 (1.1%) ActionDispatch::Journey::Formatter#generate
71 (1.1%) 71 (1.1%) ActionDispatch::Routing::RouteSet::NamedRouteCollection::UrlHelper#handle_positional_args
65 (1.0%) 65 (1.0%) ActionDispatch::Journey::Router::Utils::UriEncoder#escape
83 (1.3%) 62 (0.9%) #<Module:0x00007f36cb3a52d8>.clean_backtrace
61 (0.9%) 61 (0.9%) ActiveRecord::Result#hash_rows
58 (0.9%) 58 (0.9%) BatchLoader#__replace_with!
88 (1.3%) 55 (0.8%) #<Module:0x00007f36e32444f8>.path_for
52 (0.8%) 52 (0.8%) ActiveSupport::ToJsonWithActiveSupportEncoder#to_json
51 (0.8%) 51 (0.8%) ActiveSupport::NumericWithFormat#to_s
48 (0.7%) 48 (0.7%) Delegator.const_missing
3801 (58.0%) 48 (0.7%) ActionDispatch::Routing::RouteSet#url_for
82 (1.3%) 44 (0.7%) ActiveModel::Type::Helpers::TimeValue#new_time
57 (0.9%) 44 (0.7%) block (2 levels) in class_attribute
95 (1.4%) 43 (0.7%) ActiveSupport::JSON::Encoding::JSONGemEncoder::EscapedString#to_json
55 (0.8%) 43 (0.7%) ActiveModel::AttributeMethods#respond_to?
42 (0.6%) 42 (0.6%) ActiveRecord::Reflection::AssociationReflection#foreign_key
39 (0.6%) 39 (0.6%) Grape::Entity::Exposure::Base#attr_path
40 (0.6%) 39 (0.6%) ActiveSupport::JSON::Encoding::JSONGemEncoder#jsonify
38 (0.6%) 38 (0.6%) ActiveRecord::Transactions#update_attributes_from_transaction_state
38 (0.6%) 38 (0.6%) ActiveRecord::Base.default_timezone
37 (0.6%) 37 (0.6%) ActionDispatch::Routing::RouteSet::Generator#normalize_options!
$ stackprof pipeline.sample --method=ActionDispatch::Journey::Formatter#missing_keys
ActionDispatch::Journey::Formatter#missing_keys (/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/formatter.rb:139)
samples: 2926 self (44.6%) / 2969 total (45.3%)
callers:
2969 ( 100.0%) ActionDispatch::Journey::Formatter#generate
2951 ( 99.4%) ActionDispatch::Journey::Formatter#missing_keys
callees (43 total):
2951 ( 6862.8%) ActionDispatch::Journey::Formatter#missing_keys
27 ( 62.8%) #<Module:0x00007f36e2f696e8>.===
16 ( 37.2%) ActionDispatch::Journey::Route#required_parts