Track internal events for MLflow writes and searches

What does this MR do and why?

Adds internal events, with project, namespace and user, to the MLflow-compatible API: creating experiments and runs, setting experiment tags, logging metrics, parameters and tags to runs, and searching runs, experiments and registered models and reading metric history. Today only model and model version creation emit events, so experiment tracking usage and its growth cannot be measured or attributed to a namespace. The events do not change any response, so there is no feature flag; rollback is a revert.

Design notes
  • Events (all identifiers: [project, namespace, user], product_group: mlops, all tiers):

    Event Endpoints Additional properties
    create_mlflow_experiment experiments/create none
    create_mlflow_run runs/create none
    set_mlflow_experiment_tag experiments/set-experiment-tag none
    log_mlflow_run_data runs/log-metric, runs/log-parameter, runs/set-tag, runs/log-batch label: the endpoint (log_metric, log_parameter, set_tag, log_batch); value: number of metrics, parameters and tags submitted in the request (it can exceed the rows inserted, because duplicate parameters and tags are skipped); a log-batch without items sends no event
    search_mlflow_runs runs/search label: the order that Projects::Ml::CandidateFinder#applied_order_type reports: metric, column (a supported run attribute), or none (an unsupported order, which falls back to created_at)
    search_mlflow_experiments experiments/list, experiments/search label: list or search
    search_mlflow_registered_models registered-models/search none
    get_mlflow_metric_history metrics/get-history none
  • One event per request. log-batch sends the number of items as value instead of one event per row, so the event volume follows the request volume.

  • Each event has one counts.count_total_<event> metric for 7 and 28 days. The Snowplow events carry the project and namespace, which gives the per-namespace view.

  • API::Ml::Mlflow::ApiHelpers#track_mlflow_event wraps Gitlab::InternalEventsTracking#track_internal_event with current_user and user_project. The namespace comes from the project. Write events fire only after the write succeeds: a duplicate experiment name, an invalid metric or a duplicate parameter does not count. Search and metric history events fire after the records load, so a search that fails, for example on a statement timeout, does not count.

  • The 16 event and metric YAML files were generated with the gitlab_internal_events_cli gem's Event and Metric classes, because the interactive CLI cannot run in this environment. The output matches the CLI's.

  • No changelog entry: the change is not user-facing.

  • Overlapping sibling: !263921 (merged) changes the create endpoints of lib/api/ml/mlflow/experiments.rb and lib/api/ml/mlflow/runs.rb, and git merge-tree shows conflicts there. Whichever MR merges second rebases.

  • Possible follow-up: a sum metric over value of log_mlflow_run_data, for the number of items logged.

Deferred to follow-ups, not tracked in this MR:

  • runs/update, experiments/delete, runs/delete and the model registry update and delete endpoints: they change or remove existing rows and add no growth.
  • Single-object reads (runs/get, experiments/get, experiments/get-by-name, registered-models/get): the MLflow client calls them around most writes (for example runs/get after start_run), so their event volume needs a check first.
  • MLflow artifact listing and downloads (mlflow-artifacts): read-only access to stored package files that adds no rows; they send no event today and need their own events in a follow-up.
  • packages/ml_models uploads and downloads: already tracked by push_package_to_registry and pull_package_from_registry with the ml_model label.
  • The experiment CSV export: a web UI controller, not the MLflow-compatible API; it needs a separate controller event.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

  1. Create a personal access token with the api scope for a user with the Developer role in a project, and set TOKEN and PROJECT_ID.

  2. In a second terminal, start the event monitor:

    bin/rails runner scripts/internal_events/monitor.rb create_mlflow_experiment create_mlflow_run log_mlflow_run_data
  3. Create an experiment, then a run with the returned experiment_id:

    curl --request POST --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" \
      --data '{"name": "events-check"}' \
      "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/experiments/create"
    curl --request POST --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" \
      --data '{"experiment_id": "<experiment_id>"}' \
      "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/runs/create"
  4. Log a batch to the returned run_id:

    curl --request POST --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" \
      --data '{"run_id": "<run_id>", "metrics": [{"key": "loss", "value": 0.5, "timestamp": 1}], "params": [{"key": "lr", "value": "0.1"}]}' \
      "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/runs/log-batch"

    Expect: the monitor counts one create_mlflow_experiment, one create_mlflow_run, and one log_mlflow_run_data with label log_batch and value 2.

Other events
  • POST .../runs/search with "order_by": "metrics.loss DESC" sends search_mlflow_runs with label metric.
  • GET .../experiments/list and POST .../experiments/search send search_mlflow_experiments with label list and search.
  • GET .../registered-models/search and GET .../metrics/get-history?run_id=<run_id>&metric_key=loss send search_mlflow_registered_models and get_mlflow_metric_history.
  • Creating an experiment with an existing name returns RESOURCE_ALREADY_EXISTS and sends no event.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eduardo Bonet

Merge request reports

Loading
Loading