Bound the metric history the ML run page and CSV export load

Eduardo Bonetrequested to merge
635040-bound-ml-candidate-page-metrics into master

What does this MR do and why?

The ML run page now embeds at most the 1,000 most recent metric values of a run, and the experiment CSV export no longer loads the full metric history of every run. Today the run page puts every metric row of the run into the HTML, so the page size and render time grow with the run's history, and the CSV export runs one full-history query per run before it writes a row. Both changes are behind the ml_bounded_candidate_metric_reads feature flag; pages for runs with 1,000 metric values or fewer stay the same, and the CSV values stay the same with the metric columns sorted by name. The UI part (step columns and an alert for a truncated history) is in !264613 (merged).

Design notes
  • What the run page shows: candidate_detail.vue renders the Performance tab as a table with one row per metric name and one column per step, from every embedded metric row. There is no chart, and the page has no other request for the history.
  • Run page (app/presenters/ml/candidate_details_presenter.rb): with the flag on, #embedded_metrics loads MAX_EMBEDDED_METRICS + 1 rows with Ml::CandidateMetric.most_recent_for_candidate. A run with 1,000 rows or fewer embeds all of them, as before. A longer run embeds its 1,000 most recent rows in logging (id) order and sets metricsTruncated. 1,000 is the page size of the MLflow metrics/get-history endpoint, which stays the way to read the full history. The presenter serves the show and the promote page; the promote page does not read metrics.
  • Known limit: for a run over 1,000 rows, a metric logged only early in the run (for example a parameter count at step 0) can be missing from the table. The CSV export still has the latest value of each metric. Embedding the latest row of each metric reads and sorts every row of the run with today's indexes, so this MR leaves it out. It can come back with a loose index scan for the names and a LATERAL lookup per name once #635116 adds the (candidate_id, name, id) index (1.8 ms warm in the database review).
  • Frontend: !264613 (merged) builds the table from the logged steps only and shows an info alert when metricsTruncated is true. The two MRs merge in either order: without the frontend MR, metricsTruncated is unused; without this MR, the frontend sees no metricsTruncated and shows no alert. Merge the frontend MR before the flag is enabled. Otherwise a run over the limit shows empty columns for the steps that the page leaves out.
  • Why OFFSET 0 in most_recent_for_candidate (app/models/ml/candidate_metric.rb): with WHERE candidate_id = ? ORDER BY id DESC LIMIT 1001, PostgreSQL picks a backward scan of the primary key for a run with many rows, which can read most of the table (the rejected plan in the Database review section). OFFSET 0 stops the planner from flattening the subquery, so it uses the candidate_id index and a top-N heapsort. The same pattern is in app/models/ci/build.rb. A MATERIALIZED CTE as the fence plans slower.
  • CSV export (app/presenters/ml/candidates_csv_presenter.rb): with the flag on, the metric column names come from latest_metrics, which including_relationships already preloads and which the export already uses for the values. Each run has the same set of names in metrics and in latest_metrics, and the values were already the latest value per metric, so the values are the same. The metric columns are sorted by name. Before, they came in the order the per-run metrics query returned rows, which has no ORDER BY; that order is not stable, and it differs from the latest-row order when metrics are logged at different rates.
  • CSV row loop: with the flag on, CsvBuilder reads the rows from candidates.without_preloads (a new scope in app/models/ml/candidate.rb) and preloads only the experiment, because the headers already hold every metric and param value. Before, each batch preloaded latest_metrics, params and the other associations of including_relationships a second time. The two @candidates.to_h passes in #headers do not reload the relation: flat_map loads it once and the later passes reuse the loaded records.
  • Partly addressed: the header pass still loads every run of the experiment with its preloads before the 2 MB output cap applies. Capping the number of runs the CSV export loads is a product decision and is a follow-up.
  • The presenter takes the project as the flag actor (CandidatesCsvPresenter.new(candidates, project:)); the controller is the only caller.
  • Spec fix: the CSV N+1 spec in spec/requests/projects/ml/experiments_controller_spec.rb took its control on the first request and compared one run without metrics to two runs with metrics, so first-request queries hid the per-run query. It now takes the control on a warm second request and compares 2 to 5 runs with the same shape (2 params, 2 metrics over 3 steps). With the flag off it fails (27 to 30 queries).
  • Not changed here, for the database reviewer: the fenced query still reads every row of a long run to find the 1,000 most recent ones (12,000 to 20,000 buffers on the densest runs, the same rows read as today), but only 1,001 rows go to Ruby. An index on (candidate_id, id) without the fence would bound it, but that index is large, so it is out of scope here.
  • Sibling MRs: this MR adds one scope to app/models/ml/candidate.rb (without_preloads, after order_by_metric) and one example to spec/models/ml/candidate_spec.rb (after #latest_metrics), away from the lines that !263900 (merged), !263896 and !263939 change. !263900 (merged) and !263939 also edit spec/requests/projects/ml/experiments_controller_spec.rb (other examples), and !263939 edits app/controllers/projects/ml/experiments_controller.rb (the destroy action, a different hunk). Merge order does not matter; whoever lands second rebases.

References

Feature flag and rollback

  • ml_bounded_candidate_metric_reads (gitlab_com_derisk, default off, project actor) switches the run page and the CSV export together. Rollback: disable the flag.
  • Enable the flag only after !264613 (merged) is merged.
  • No migration.

Screenshots or screen recordings

The UI change and its screenshots are in !264613 (merged). This MR adds the metricsTruncated field to the page data and limits the embedded rows when the flag is on; with the flag off, the page is unchanged.

How to set up and validate locally

  1. In the rails console, create a run with 1,201 metric values (a parameter count at step 0, then 2 metrics over 600 steps) and a run with 6 values:

    project = Project.find_by_full_path('flightjs/Flight'); user = User.find_by_username('root')
exp = Ml::Experiment.create!(name: 'bounded-metrics', project: project, user: user)
long = exp.candidates.create!(project: project, user: user); long.metrics.create!(name: 'num_params', value: 1000, step: 0, tracked_at: 0); 600.times { |s| long.metrics.create!([{ name: 'loss', value: 1.0 / (s + 1), step: s, tracked_at: s }, { name: 'accuracy', value: s / 600.0, step: s, tracked_at: s }]) }
short = exp.candidates.create!(project: project, user: user); 3.times { |s| short.metrics.create!([{ name: 'loss', value: s, step: s, tracked_at: s }, { name: 'accuracy', value: s, step: s, tracked_at: s }]) }
[exp.iid, long.iid, short.iid]

  2. Visit http://gdk.test:3000/flightjs/Flight/-/ml/candidates/<long iid> and open the Performance tab. Expect: columns Step 0 to Step 599 with values.

  3. Enable the flag: Feature.enable(:ml_bounded_candidate_metric_reads).

  4. Reload the page. Expect: values only for Step 100 to Step 599, with the earlier step columns empty, and no num_params row, because the run is over the limit. With !264613 (merged) also checked out, the table has only the columns Step 100 to Step 599 and an info alert.

  5. Visit the page of the short run. Expect: the same table as with the flag off (Step 0 to Step 2).

  6. Visit http://gdk.test:3000/flightjs/Flight/-/ml/experiments/<exp iid>.csv with the flag on and off. Expect: the same values, with the accuracy and loss columns holding the latest values (sorted by name with the flag on). log/development.log shows no ... FROM "ml_candidate_metrics" WHERE "ml_candidate_metrics"."candidate_id" = <id> query per run with the flag on.

Specs that cover the change
  • spec/requests/projects/ml/candidates_controller_spec.rb: an anonymous user on a public project; the page embeds only the most recent rows of a long run (a metric logged only at the start is left out), and every row of a short run, with the flag on and off.
  • spec/requests/projects/ml/experiments_controller_spec.rb: the CSV N+1 spec with a warm control.
  • spec/presenters/ml/candidates_csv_presenter_spec.rb: metrics logged at different rates; latest values with the columns sorted by name, no per-run metric query of any shape, the same values with the flag on and off.
  • spec/presenters/ml/candidate_details_presenter_spec.rb: database rows at the limit (1,000, not truncated) and one above (1,001, truncated), flag on and off.
  • spec/models/ml/candidate_metric_spec.rb: most_recent_for_candidate returns the newest rows of one run when rows of two runs interleave.
  • spec/models/ml/candidate_spec.rb: without_preloads drops includes and preload.

Local measurements (request spec, anonymous, one run, metrics over 10 names):

Metric rows in the run Flag off Flag on
10 35 KB 35 KB
1,500 547 KB 373 KB
100,000 34.6 MB, 12-15 s 378 KB, 1.1 s

CSV export with 100 metric rows per run: 2 runs 25 queries, 20 runs 43 queries with the flag off; 19 queries for both with the flag on, with one ml_candidate_metrics query.

Database review

This MR changes the run page query, and removes one query per run and the second latest_metrics preload from the CSV export, behind the flag. It has no migration. ml_candidate_metrics is 7.5 GB on GitLab.com and has indexes on id, candidate_id and project_id.

Query Flag off (today) Flag on (this MR)
Run page, run with the most rows (614,560) 614,560 rows to Ruby, 125 ms warm, 12,428 buffers 1,001 rows, 142 ms warm (2.4 s cold), 12,431 buffers, top-N heapsort 297 kB
Run page, run with 394 rows 394 rows 394 rows, 0.6 ms, 144 buffers
Run page, run with 34 rows 34 rows 34 rows, 0.1 ms, 13 buffers
CSV export, per run one SELECT * ... WHERE candidate_id = ? per run, returning its full history none
CSV export, row loop latest_metrics, params and four more preloads per batch of 1,000 runs experiment only
Query plans (Database Lab, gitlab-production-main)

Run page, flag on:

SELECT "ml_candidate_metrics".* FROM (SELECT "ml_candidate_metrics".* FROM "ml_candidate_metrics" WHERE "ml_candidate_metrics"."candidate_id" = <run_id> OFFSET 0) ml_candidate_metrics ORDER BY "ml_candidate_metrics"."id" DESC LIMIT 1001

Run with the most rows, warm:

 Limit  (cost=84233.01..84235.51 rows=1001 width=114) (actual time=141.777..141.904 rows=1001 loops=1)
   Buffers: shared hit=12431
   ->  Sort  (cost=84233.01..85698.33 rows=586130 width=114) (actual time=141.776..141.829 rows=1001 loops=1)
         Sort Key: ml_candidate_metrics.id DESC
         Sort Method: top-N heapsort  Memory: 297kB
         Buffers: shared hit=12431
         ->  Index Scan using index_ml_candidate_metrics_on_candidate_id on public.ml_candidate_metrics  (cost=0.44..52091.91 rows=586130 width=114) (actual time=0.029..86.757 rows=614560 loops=1)
               Index Cond: (ml_candidate_metrics.candidate_id = <run_id>)
               Buffers: shared hit=12428
Execution Time: 142.009 ms

The same run, cold: 2.397 s, shared hit=282 read=12149, same plan.

Run with 394 rows:

 Limit  (cost=29.90..30.43 rows=210 width=114) (actual time=0.471..0.526 rows=394 loops=1)
   Buffers: shared hit=144
   ->  Sort  (cost=29.90..30.43 rows=210 width=114) (actual time=0.471..0.493 rows=394 loops=1)
         Sort Key: ml_candidate_metrics.id DESC
         Sort Method: quicksort  Memory: 107kB
         Buffers: shared hit=144
         ->  Index Scan using index_ml_candidate_metrics_on_candidate_id on public.ml_candidate_metrics  (cost=0.44..21.80 rows=210 width=114) (actual time=0.031..0.352 rows=394 loops=1)
               Index Cond: (ml_candidate_metrics.candidate_id = <run_id>)
               Buffers: shared hit=141
Execution Time: 0.590 ms

Run with 34 rows:

 Limit  (cost=29.90..30.43 rows=210 width=114) (actual time=0.073..0.078 rows=34 loops=1)
   Buffers: shared hit=13
   ->  Sort  (cost=29.90..30.43 rows=210 width=114) (actual time=0.072..0.074 rows=34 loops=1)
         Sort Key: ml_candidate_metrics.id DESC
         Sort Method: quicksort  Memory: 29kB
         Buffers: shared hit=13
         ->  Index Scan using index_ml_candidate_metrics_on_candidate_id on public.ml_candidate_metrics  (cost=0.44..21.80 rows=210 width=114) (actual time=0.035..0.051 rows=34 loops=1)
               Index Cond: (ml_candidate_metrics.candidate_id = <run_id>)
               Buffers: shared hit=10
Execution Time: 0.130 ms

Run page, flag off (today), run with the most rows:

SELECT "ml_candidate_metrics".* FROM "ml_candidate_metrics" WHERE "ml_candidate_metrics"."candidate_id" = <run_id>
 Index Scan using index_ml_candidate_metrics_on_candidate_id on public.ml_candidate_metrics  (cost=0.44..52091.91 rows=586130 width=114) (actual time=0.032..98.484 rows=614560 loops=1)
   Index Cond: (ml_candidate_metrics.candidate_id = <run_id>)
   Buffers: shared hit=12428
Execution Time: 125.153 ms

The CSV export ran the same query once per run of the experiment; with the flag on it does not run.

CSV export row loop, flag on. CsvBuilder#each runs the same each_batch queries as on master:

SELECT "ml_candidates"."id" FROM "ml_candidates" WHERE "ml_candidates"."experiment_id" = <experiment_id> ORDER BY "ml_candidates"."id" ASC, "ml_candidates"."created_at" ASC LIMIT 1
SELECT "ml_candidates"."id" FROM "ml_candidates" WHERE "ml_candidates"."experiment_id" = <experiment_id> AND "ml_candidates"."id" >= <start_id> ORDER BY "ml_candidates"."id" ASC, "ml_candidates"."created_at" ASC LIMIT 1 OFFSET 1000
SELECT "ml_candidates".* FROM "ml_candidates" WHERE "ml_candidates"."experiment_id" = <experiment_id> AND "ml_candidates"."id" >= <start_id> ORDER BY "ml_candidates"."id" ASC

With without_preloads, a batch no longer preloads latest_metrics, params, user, project, package and ci_build. The remaining .preload(:experiment) runs no query, because experiment.candidates already sets the experiment through the inverse association.

Rejected shape, without OFFSET 0, run with the most rows:

SELECT "ml_candidate_metrics".* FROM "ml_candidate_metrics" WHERE "ml_candidate_metrics"."candidate_id" = <run_id> ORDER BY "ml_candidate_metrics"."id" DESC LIMIT 1001
 Limit  (cost=0.56..6589.82 rows=1001 width=114) (actual time=293225.660..293237.409 rows=1001 loops=1)
   Buffers: shared hit=7369229 read=571932 dirtied=39468 written=19874
   ->  Index Scan Backward using ml_candidate_metrics_pkey on public.ml_candidate_metrics  (cost=0.56..3858300.85 rows=586130 width=114) (actual time=293225.659..293237.329 rows=1001 loops=1)
         Filter: (ml_candidate_metrics.candidate_id = <run_id>)
         Rows Removed by Filter: 32798695
Execution Time: 4.887 min

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eduardo Bonet

Merge request reports