Track internal events for MLflow writes and searches
What does this MR do and why?
Adds internal events, with project, namespace and user, to the MLflow-compatible API: creating experiments and runs, setting experiment tags, logging metrics, parameters and tags to runs, and searching runs, experiments and registered models and reading metric history. Today only model and model version creation emit events, so experiment tracking usage and its growth cannot be measured or attributed to a namespace. The events do not change any response, so there is no feature flag; rollback is a revert.
Design notes
-
Events (all
identifiers: [project, namespace, user],product_group: mlops, all tiers):Event Endpoints Additional properties create_mlflow_experimentexperiments/createnone create_mlflow_runruns/createnone set_mlflow_experiment_tagexperiments/set-experiment-tagnone log_mlflow_run_dataruns/log-metric,runs/log-parameter,runs/set-tag,runs/log-batchlabel: the endpoint (log_metric,log_parameter,set_tag,log_batch);value: number of metrics, parameters and tags submitted in the request (it can exceed the rows inserted, because duplicate parameters and tags are skipped); alog-batchwithout items sends no eventsearch_mlflow_runsruns/searchlabel: the order thatProjects::Ml::CandidateFinder#applied_order_typereports:metric,column(a supported run attribute), ornone(an unsupported order, which falls back tocreated_at)search_mlflow_experimentsexperiments/list,experiments/searchlabel:listorsearchsearch_mlflow_registered_modelsregistered-models/searchnone get_mlflow_metric_historymetrics/get-historynone -
One event per request.
log-batchsends the number of items asvalueinstead of one event per row, so the event volume follows the request volume. -
Each event has one
counts.count_total_<event>metric for 7 and 28 days. The Snowplow events carry the project and namespace, which gives the per-namespace view. -
API::Ml::Mlflow::ApiHelpers#track_mlflow_eventwrapsGitlab::InternalEventsTracking#track_internal_eventwithcurrent_useranduser_project. The namespace comes from the project. Write events fire only after the write succeeds: a duplicate experiment name, an invalid metric or a duplicate parameter does not count. Search and metric history events fire after the records load, so a search that fails, for example on a statement timeout, does not count. -
The 16 event and metric YAML files were generated with the
gitlab_internal_events_cligem'sEventandMetricclasses, because the interactive CLI cannot run in this environment. The output matches the CLI's. -
No changelog entry: the change is not user-facing.
-
Overlapping sibling: !263921 (merged) changes the
createendpoints oflib/api/ml/mlflow/experiments.rbandlib/api/ml/mlflow/runs.rb, andgit merge-treeshows conflicts there. Whichever MR merges second rebases. -
Possible follow-up: a sum metric over
valueoflog_mlflow_run_data, for the number of items logged.
Deferred to follow-ups, not tracked in this MR:
runs/update,experiments/delete,runs/deleteand the model registry update and delete endpoints: they change or remove existing rows and add no growth.- Single-object reads (
runs/get,experiments/get,experiments/get-by-name,registered-models/get): the MLflow client calls them around most writes (for exampleruns/getafterstart_run), so their event volume needs a check first. - MLflow artifact listing and downloads (
mlflow-artifacts): read-only access to stored package files that adds no rows; they send no event today and need their own events in a follow-up. packages/ml_modelsuploads and downloads: already tracked bypush_package_to_registryandpull_package_from_registrywith theml_modellabel.- The experiment CSV export: a web UI controller, not the MLflow-compatible API; it needs a separate controller event.
References
Screenshots or screen recordings
No UI change.
How to set up and validate locally
-
Create a personal access token with the
apiscope for a user with the Developer role in a project, and setTOKENandPROJECT_ID. -
In a second terminal, start the event monitor:
bin/rails runner scripts/internal_events/monitor.rb create_mlflow_experiment create_mlflow_run log_mlflow_run_data -
Create an experiment, then a run with the returned
experiment_id:curl --request POST --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" \ --data '{"name": "events-check"}' \ "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/experiments/create" curl --request POST --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" \ --data '{"experiment_id": "<experiment_id>"}' \ "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/runs/create" -
Log a batch to the returned
run_id:curl --request POST --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/json" \ --data '{"run_id": "<run_id>", "metrics": [{"key": "loss", "value": 0.5, "timestamp": 1}], "params": [{"key": "lr", "value": "0.1"}]}' \ "http://gdk.test:3000/api/v4/projects/$PROJECT_ID/ml/mlflow/api/2.0/mlflow/runs/log-batch"Expect: the monitor counts one
create_mlflow_experiment, onecreate_mlflow_run, and onelog_mlflow_run_datawithlabellog_batchandvalue2.
Other events
POST .../runs/searchwith"order_by": "metrics.loss DESC"sendssearch_mlflow_runswithlabelmetric.GET .../experiments/listandPOST .../experiments/searchsendsearch_mlflow_experimentswithlabellistandsearch.GET .../registered-models/searchandGET .../metrics/get-history?run_id=<run_id>&metric_key=losssendsearch_mlflow_registered_modelsandget_mlflow_metric_history.- Creating an experiment with an existing name returns
RESOURCE_ALREADY_EXISTSand sends no event.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.