Adds MlExperimentMetadata and MlCandidateMetada

What does this MR do and why?

MLFlow Clients sends metadata about runs and experiments that we are ignoring. With this MR we start storing them. Tags are set once only, both for experiments and candidates, and are returned when fetching a candidate or experiment.

Database

Migrations

Up

❯ bin/rails db:migrate RAILS_ENV=test
main: == 20221116160204 CreateMlExperimentMetadataAndMlCandidateMetadata: migrating =
main: -- create_table(:ml_experiment_metadata, {})
main: -- quote_column_name(:name)
main:    -> 0.0000s
main: -- quote_column_name(:value)
main:    -> 0.0000s
main:    -> 0.0289s
main: -- create_table(:ml_candidate_metadata, {})
main: -- quote_column_name(:name)
main:    -> 0.0000s
main: -- quote_column_name(:value)
main:    -> 0.0000s
main:    -> 0.0157s
main: == 20221116160204 CreateMlExperimentMetadataAndMlCandidateMetadata: migrated (0.0459s)

ci: == 20221116160204 CreateMlExperimentMetadataAndMlCandidateMetadata: migrating =
ci: -- create_table(:ml_experiment_metadata, {})
ci: -- quote_column_name(:name)
ci:    -> 0.0000s
ci: -- quote_column_name(:value)
ci:    -> 0.0000s
ci:    -> 0.0273s
ci: -- create_table(:ml_candidate_metadata, {})
ci: -- quote_column_name(:name)
ci:    -> 0.0000s
ci: -- quote_column_name(:value)
ci:    -> 0.0000s
ci:    -> 0.0118s
ci: == 20221116160204 CreateMlExperimentMetadataAndMlCandidateMetadata: migrated (0.0394s)

Down

❯ bundle exec rails db:migrate:down:main RAILS_ENV=test VERSION=20221116160204
main: == 20221116160204 CreateMlExperimentMetadataAndMlCandidateMetadata: reverting =
main: -- drop_table(:ml_candidate_metadata, {})
main:    -> 0.0238s
main: -- drop_table(:ml_experiment_metadata, {})
main:    -> 0.0044s
main: == 20221116160204 CreateMlExperimentMetadataAndMlCandidateMetadata: reverted (0.0324s)

Queries

Fetching @candidate.metadata:

SELECT "ml_candidate_metadata".* FROM "ml_candidate_metadata" WHERE "ml_candidate_metadata"."candidate_id" = 25 

Bitmap Heap Scan on ml_candidate_metadata  (cost=4.17..11.28 rows=3 width=96)
  Recheck Cond: (candidate_id = 25)
  ->  Bitmap Index Scan on index_ml_candidate_metadata_on_candidate_id_and_name  (cost=0.00..4.17 rows=3 width=0)
        Index Cond: (candidate_id = 25)

Fetching @experiment.metadata

SELECT "ml_experiment_metadata".* FROM "ml_experiment_metadata" WHERE "ml_experiment_metadata"."experiment_id" = 12

Bitmap Heap Scan on ml_experiment_metadata  (cost=4.17..11.28 rows=3 width=96)
  Recheck Cond: (experiment_id = 12)
  ->  Bitmap Index Scan on index_ml_experiment_metadata_on_experiment_id_and_name  (cost=0.00..4.17 rows=3 width=0)
        Index Cond: (experiment_id = 12)

How to set up and validate locally

1. Enable the Feature flag

   echo "Feature.enable(:ml_experiment_tracking)" | bundle exec rails c

2. Create a Project and an personal access token:

   export PROJECT_ID=<Your Project Id>
   export GITLAB_PAT=<your api token>

3. Create an Experiment:

   curl -X POST -H "Authorization: Bearer $GITLAB_PAT" \
       -H "Content-Type: application/json"  \
       -d '{"name":"some_experiment","tags":[{"key":"a", "value":"b"}]}' \
       http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/experiments/create

4. Get the Experiment, notice tags is now added with this MR

   curl -X GET -H "Authorization: Bearer $GITLAB_PAT" "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/experiments/get?experiment_id=1"

5. Create a Run, notice response contains now tags is now added with this MR

   curl -X POST -H "Authorization: Bearer $GITLAB_PAT" -H "Content-Type: application/json" --data '{"experiment_id":1,"tags":[{"key":"a", "value":"b"}]}' http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/runs/create

   

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #374928 (closed)