Make new machine learning runs use ml_model package type

Context

We aim to make new machine learning runs (ml_candidates) use ml_model package type instead of the generic packages. We should also migrate existing records. In the model registry the model version packages already use the ml_model type already.

See Merge Model experiments into Model registry for background.

History

Existing candidates use the generic package type to store artifacts, while model registry uses the new ml model type and its endpoints. These endpoints allow more domain control over the creation of packages, and should simplify the logic behind storing candidate artifacts. We can keep existing candidates using the generic package, or find a way to migrate them to the new ml model package type.

As of Feb 2024, gitab.com database has ~38k candidates out of which ~4k have packages that need to be migrated. We don’t have much data on self managed instances, but hte migration should work for them as well. Iteration 1: New candidates use the ml_model package type

This is already being done for candidates that are part of a model_version. For others, when we identify the model registry feature flag is enabled we could already support candidate by adding a new endpoint to ml_model_packages.rb Iteration 2: Packages for existing candidates to be migrated to ml_model

Add migration to updates all generic packages associated to a candidate to package_type ml_model. These packages follow a different naming convention (ml_experiment_{experiment_iid}/{candidate_iid}) than that supported by ml_model packages ({ml_model_name}/{semver_version}), but since for now they will still belong to an experiment we should allow ml_models to accept the existing name and version.

Proposal

1. Make new ML candidates use ml_model package type
2. Ensure existing ML candidates with generic package type work fine in a dual logic