Limit ML candidate metrics, params and metadata to one candidate

What does this MR do and why?

The metrics, params and metadata fields of MlCandidate now resolve for one candidate per GraphQL operation, behind the limit_ml_candidate_graphql_fields feature flag (gitlab_com_derisk, default off, project actor). Each of these connections runs its own query for every candidate, and their complexity does not grow with the page size, so one query through mlExperiment.candidates or mlModel.versions runs three queries per candidate it returns. The limit is Gitlab::Graphql::Limit::FieldCallCount with limit: 1, as on MlModel.versions; rollback is to disable the flag.

Design notes
  • Why the child fields and not candidates: a limit on MlExperiment.candidates or MlModel.candidates misses mlModel { versions(first: 100) { nodes { candidate { metrics { ... } } } } } (the versions limit is per model, so each version's candidate runs its own three queries). It also breaks API clients that list candidates without these three fields.
  • The flag check is in lib/gitlab/graphql/limit/ml_candidate_field_call_count.rb, a subclass of FieldCallCount that skips the count when the flag is off for the candidate's project. The actor comes from Project.actor_from_id(candidate.project_id), so the check loads no record. After the rollout, the three fields use FieldCallCount directly and the subclass is deleted (cleanup step of the rollout issue).
  • Response over the limit: FieldCallCount raises LimitError with "metrics" field can be requested only for 1 MlCandidate(s) at a time.. The three fields are non-null, so the error sets the nearest nullable parent to null. Through candidates, that is the candidate node. Through versions, MlModelVersion.candidate is also non-null, so the whole version node becomes null, including version; the same applies to mlModel.version(...) and latestVersion when their candidate is over the limit. The first candidate resolves as before. graphql-ruby reports the first failing field of each node.
  • The field descriptions name the flag and the limit, as the GraphQL style guide asks for feature-flagged behavior and for FieldCallCount. The GraphQL reference and the introspection files are regenerated.
  • Query count: with the flag on, a query reads ml_candidate_metrics, ml_candidate_params and ml_candidate_metadata once each, whatever the number of candidates. The spec in spec/requests/api/graphql/ml/experiments_spec.rb counts the queries per table.
  • Follow-ups, not in this MR:
    • With the flag on, the other candidate fields still cost work per candidate. creator, ciJob and _links { showPath } look up records for each candidate (several uncached queries per candidate in the test environment), so a query that lists many candidates with these fields is still not bounded by one page. Batch those presenter lookups, or bound the nested candidates connection.
    • Batch the metrics, params and metadata connections across candidates, which could let the limit go later.

Breaking change

A query that selects one of these fields for more than one candidate returns an error with the flag on, and loses data for every candidate after the first. API clients that list model versions with candidate metrics, params or metadata lose whole version entries, not only the candidate fields. The GraphQL style guide counts "anything else that can result in queries hitting a limit that previously was allowed" as a breaking change. The documented exemption for performance concerns applies instead of the deprecation process: "GitLab might make immediate breaking changes to the GraphQL API to patch critical security or performance concerns if the deprecation process would pose significant risk" (Breaking change exemptions). The deprecation process would keep the per-candidate cost until the next major release. The flag stays off on GitLab.com until the change has approval through the Breaking Change Exception process; the request is https://gitlab.com/gitlab-com/Product/-/work_items/14590, and the rollout issue tracks it. The flag gives a rollback.

The limit counts calls of each field, not candidates (existing FieldCallCount behavior): one candidate with m1: metrics(first: 1) m2: metrics(first: 2) also fails on m2.

The UI is not affected. Only getModel and getModelVersion (model and model version detail pages) select these fields, for one candidate each, and the new specs run both query files. No other frontend query and no internal GraphQL caller selects them. GitLab.com GraphQL logs show a small number of API requests that select them for several candidates. API clients that select them for several candidates in one query need one query per candidate.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. In a Rails console, enable the flag: Feature.enable(:limit_ml_candidate_graphql_fields).

  2. In any project, go to Deploy > Model registry, select Create model, and create a model.

  3. On the model page, select Create new version and create version 1.0.0. Repeat for version 1.0.1.

  4. Open the model page, then the page of version 1.0.0. Expect: both pages load as on master, with no error message.

  5. Open http://gdk.test:3000/-/graphql-explorer and run the query below with the model ID from the page URL. Expect: versions.nodes has one version with its version and its candidate's metrics, and one null entry in place of the other version. errors has "metrics" field can be requested only for 1 MlCandidate(s) at a time.

    query {
      mlModel(id: "gid://gitlab/Ml::Model/<model_id>") {
        versions { nodes { version candidate { id metrics { nodes { name value } } } } }
      }
    }
  6. Run Feature.disable(:limit_ml_candidate_graphql_fields) and run the query again. Expect: both versions with their candidates, no error.

More scenarios
  • Experiments, flag on: mlExperiment(id: "gid://gitlab/Ml::Experiment/<id>") { candidates { nodes { id params { nodes { name } } } } } on an experiment with two or more candidates returns the params of one candidate, null for the others, and the "params" limit error.
  • metadata behaves the same.
  • A query that selects metrics, params and metadata for one candidate, for example mlModel { latestVersion { candidate { ... } } }, returns no error.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Database review

Not needed: the change adds and changes no query, index or migration. It stops queries from running for the second and later candidates.

Edited by Eduardo Bonet

Merge request reports

Loading
Loading