Issues with normalized_mutual_info_score metric
Two pipelines failed CI due to the following FutureWarning. The CI job log can be found at this link. Both pipelines work on the clustering problem. Pipeline josn and meta file can be found at link1 and link2.
ERROR:redirect:/usr/local/lib/python3.6/dist-packages/sklearn/metrics/cluster/supervised.py:844: FutureWarning: The behavior of NMI will change in version 0.22. To match the behavior of 'v_measure_score', NMI will use average_method='arithmetic' by default.
ERROR:redirect: FutureWarning)
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 878, in _do_run_step
self._run_step(step)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 868, in _run_step
self._run_primitive(step)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 776, in _run_primitive
multi_call_result = self._call_primitive_method(primitive.fit_multi_produce, fit_multi_produce_arguments)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 851, in _call_primitive_method
raise error
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 847, in _call_primitive_method
result = method(**arguments)
File "/src/common-primitives/common_primitives/compute_scores.py", line 149, in fit_multi_produce
return self._fit_multi_produce(produce_methods=produce_methods, timeout=timeout, iterations=iterations, inputs=inputs, score_dataset=score_dataset)
File "/usr/local/lib/python3.6/dist-packages/d3m/primitive_interfaces/base.py", line 556, in _fit_multi_produce
produce_results = self._multi_produce(produce_methods=produce_methods, timeout=timeout, iterations=iterations, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/d3m/primitive_interfaces/base.py", line 452, in _multi_produce
results.append(getattr(self, method_name)(timeout=timeout, iterations=iterations, **arguments))
File "/src/common-primitives/common_primitives/compute_scores.py", line 125, in produce
score = metric.get_class()(**params).score(truth, predictions)
File "/usr/local/lib/python3.6/dist-packages/d3m/metrics.py", line 279, in score
return metrics.normalized_mutual_info_score(truth, predictions)
File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/cluster/supervised.py", line 846, in normalized_mutual_info_score
labels_true, labels_pred = check_clusterings(labels_true, labels_pred)
File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/cluster/supervised.py", line 53, in check_clusterings
"labels_true must be 1D: shape is %r" % (labels_true.shape,))
ValueError: labels_true must be 1D: shape is (1600, 1)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 2723, in <module>
main(sys.argv)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 2719, in main
handler(arguments, parser)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 2287, in handler
arguments.runtime_handler(arguments, pipeline_resolver=pipeline_resolver, meta_parser=meta_parser, dataset_resolver=dataset_resolver, problem_resolver=problem_resolver)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 2058, in _fit_score
score_result.check_success()
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 58, in check_success
raise self.error
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 909, in _run
self._do_run()
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 895, in _do_run
self._do_run_step(step)
File "/usr/local/lib/python3.6/dist-packages/d3m/runtime.py", line 887, in _do_run_step
) from error
Edited by Mitar