ISI: pipeline runs fail in pipelines CI
Your pipeline runs seems to fail in pipelines CI. Care to fix them?
https://gitlab.datadrivendiscovery.org/d3m/pipelines-ci/-/jobs/53632
primitives/v2020.1.9/ISI/d3m.primitives.data_preprocessing.time_series_to_list.DSBOX/1.5.3/pipeline_runs/pipeline_run_66_chlorineConcentration_MIN_METADATA_1.yaml.gz
[2020-03-11 00:40:55,633] Traceback (most recent call last):
[2020-03-11 00:40:55,634] File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[2020-03-11 00:40:55,634] "__main__", mod_spec)
[2020-03-11 00:40:55,634] File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
[2020-03-11 00:40:55,634] exec(code, run_globals)
[2020-03-11 00:40:55,634] File "/src/d3m/d3m/__main__.py", line 6, in <module>
[2020-03-11 00:40:55,634] cli.main(sys.argv)
[2020-03-11 00:40:55,634] File "/src/d3m/d3m/cli.py", line 1111, in main
[2020-03-11 00:40:55,634] handler(arguments, parser)
[2020-03-11 00:40:55,634] File "/src/d3m/d3m/cli.py", line 1026, in handler
[2020-03-11 00:40:55,634] problem_resolver=problem_resolver,
[2020-03-11 00:40:55,635] File "/src/d3m/d3m/cli.py", line 516, in runtime_handler
[2020-03-11 00:40:55,635] problem_resolver=problem_resolver,
[2020-03-11 00:40:55,635] File "/src/d3m/d3m/runtime.py", line 2477, in fit_score_handler
[2020-03-11 00:40:55,635] save_steps_outputs(produce_result, arguments.expose_produced_outputs_dir)
[2020-03-11 00:40:55,635] File "/src/d3m/d3m/runtime.py", line 2796, in save_steps_outputs
[2020-03-11 00:40:55,635] container_utils.save_container(step_output, os.path.join(output_dir, key))
[2020-03-11 00:40:55,635] File "/src/d3m/d3m/container/utils.py", line 25, in save_container
[2020-03-11 00:40:55,635] raise exceptions.NotSupportedError("Value with type '{value_type}' cannot be saved as a container type.".format(value_type=type(container)))
[2020-03-11 00:40:55,636] d3m.exceptions.NotSupportedError: Value with type '<class 'list'>' cannot be saved as a container type.
primitives/v2020.1.9/ISI/d3m.primitives.feature_extraction.random_projection_timeseries_featurization.DSBOX/1.5.3/pipeline_runs/pipeline_run_66_chlorineConcentration_MIN_METADATA_1.yaml.gz
[2020-03-11 00:41:17,366] Traceback (most recent call last):
[2020-03-11 00:41:17,366] File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[2020-03-11 00:41:17,366] "__main__", mod_spec)
[2020-03-11 00:41:17,366] File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
[2020-03-11 00:41:17,366] exec(code, run_globals)
[2020-03-11 00:41:17,366] File "/src/d3m/d3m/__main__.py", line 6, in <module>
[2020-03-11 00:41:17,367] cli.main(sys.argv)
[2020-03-11 00:41:17,367] File "/src/d3m/d3m/cli.py", line 1111, in main
[2020-03-11 00:41:17,367] handler(arguments, parser)
[2020-03-11 00:41:17,367] File "/src/d3m/d3m/cli.py", line 1026, in handler
[2020-03-11 00:41:17,367] problem_resolver=problem_resolver,
[2020-03-11 00:41:17,367] File "/src/d3m/d3m/cli.py", line 516, in runtime_handler
[2020-03-11 00:41:17,367] problem_resolver=problem_resolver,
[2020-03-11 00:41:17,367] File "/src/d3m/d3m/runtime.py", line 2477, in fit_score_handler
[2020-03-11 00:41:17,367] save_steps_outputs(produce_result, arguments.expose_produced_outputs_dir)
[2020-03-11 00:41:17,368] File "/src/d3m/d3m/runtime.py", line 2796, in save_steps_outputs
[2020-03-11 00:41:17,368] container_utils.save_container(step_output, os.path.join(output_dir, key))
[2020-03-11 00:41:17,368] File "/src/d3m/d3m/container/utils.py", line 25, in save_container
[2020-03-11 00:41:17,368] raise exceptions.NotSupportedError("Value with type '{value_type}' cannot be saved as a container type.".format(value_type=type(container)))
[2020-03-11 00:41:17,368] d3m.exceptions.NotSupportedError: Value with type '<class 'list'>' cannot be saved as a container type.
v2020.1.9/ISI/d3m.primitives.data_transformation.to_numeric.DSBOX/1.5.3/pipeline_runs/pipeline_run_22_handgeometry_MIN_METADATA_3.yaml.gz
[2020-03-11 03:10:01,702] Traceback (most recent call last):
[2020-03-11 03:10:01,702] File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[2020-03-11 03:10:01,702] "__main__", mod_spec)
[2020-03-11 03:10:01,702] File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
[2020-03-11 03:10:01,702] exec(code, run_globals)
[2020-03-11 03:10:01,703] File "/src/d3m/d3m/__main__.py", line 6, in <module>
[2020-03-11 03:10:01,703] cli.main(sys.argv)
[2020-03-11 03:10:01,703] File "/src/d3m/d3m/cli.py", line 1111, in main
[2020-03-11 03:10:01,703] handler(arguments, parser)
[2020-03-11 03:10:01,703] File "/src/d3m/d3m/cli.py", line 1026, in handler
[2020-03-11 03:10:01,703] problem_resolver=problem_resolver,
[2020-03-11 03:10:01,703] File "/src/d3m/d3m/cli.py", line 516, in runtime_handler
[2020-03-11 03:10:01,703] problem_resolver=problem_resolver,
[2020-03-11 03:10:01,704] File "/src/d3m/d3m/runtime.py", line 2477, in fit_score_handler
[2020-03-11 03:10:01,704] save_steps_outputs(produce_result, arguments.expose_produced_outputs_dir)
[2020-03-11 03:10:01,704] File "/src/d3m/d3m/runtime.py", line 2796, in save_steps_outputs
[2020-03-11 03:10:01,704] container_utils.save_container(step_output, os.path.join(output_dir, key))
[2020-03-11 03:10:01,704] File "/src/d3m/d3m/container/utils.py", line 25, in save_container
[2020-03-11 03:10:01,704] raise exceptions.NotSupportedError("Value with type '{value_type}' cannot be saved as a container type.".format(value_type=type(container)))
[2020-03-11 03:10:01,704] d3m.exceptions.NotSupportedError: Value with type '<class 'list'>' cannot be saved as a container type.
Also see the following warning:
[2020-03-11 03:09:41,550] 2020-03-11 03:09:41.549690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10481 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
[2020-03-11 03:09:46,689] 2020-03-11 03:09:46.688228: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open /volumes/7011d39ea4f61f4ddb8da99c4addf3fae4209bfda7828adb4698b16283258fbe: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
v2020.1.9/ISI/d3m.primitives.feature_construction.gcn_mixhop.DSBOX/1.0.0/pipeline_runs/LL1_VTXC_1369_synthetic_MIN_METADATA_pipeline_run.yml.gz
[2020-03-11 08:15:05,862] Traceback (most recent call last):
[2020-03-11 08:15:05,862] File "/src/d3m/d3m/runtime.py", line 941, in _do_run_step
[2020-03-11 08:15:05,862] self._run_step(step)
[2020-03-11 08:15:05,862] File "/src/d3m/d3m/runtime.py", line 931, in _run_step
[2020-03-11 08:15:05,862] self._run_primitive(step)
[2020-03-11 08:15:05,862] File "/src/d3m/d3m/runtime.py", line 839, in _run_primitive
[2020-03-11 08:15:05,863] multi_call_result = self._call_primitive_method(primitive.fit_multi_produce, fit_multi_produce_arguments)
[2020-03-11 08:15:05,863] File "/src/d3m/d3m/runtime.py", line 914, in _call_primitive_method
[2020-03-11 08:15:05,863] raise error
[2020-03-11 08:15:05,863] File "/src/d3m/d3m/runtime.py", line 910, in _call_primitive_method
[2020-03-11 08:15:05,863] result = method(**arguments)
[2020-03-11 08:15:05,863] File "/src/dsbox-graphs/dsbox_graphs/gcn_mixhop.py", line 903, in fit_multi_produce
[2020-03-11 08:15:05,863] return self._fit_multi_produce(produce_methods=produce_methods, timeout=timeout, iterations=iterations, inputs=inputs, outputs=outputs)
[2020-03-11 08:15:05,863] File "/src/d3m/d3m/primitive_interfaces/base.py", line 545, in _fit_multi_produce
[2020-03-11 08:15:05,863] self.set_training_data(**arguments) # type: ignore
[2020-03-11 08:15:05,864] File "/src/dsbox-graphs/dsbox_graphs/gcn_mixhop.py", line 420, in set_training_data
[2020-03-11 08:15:05,864] incl_adj = self.hyperparams['include_adjacency'])
[2020-03-11 08:15:05,864] File "/src/dsbox-graphs/dsbox_graphs/gcn_mixhop.py", line 606, in _make_input_features
[2020-03-11 08:15:05,864] features = get_columns_of_type(nodes_df, semantic_types).values.astype(np.float32)
[2020-03-11 08:15:05,864] File "/src/dsbox-graphs/dsbox_graphs/gcn_mixhop.py", line 186, in get_columns_of_type
[2020-03-11 08:15:05,864] return df.select_columns(columns_to_use)
[2020-03-11 08:15:05,864] File "/src/d3m/d3m/container/pandas.py", line 220, in select_columns
[2020-03-11 08:15:05,864] output = self.iloc[:, list(columns)]
[2020-03-11 08:15:05,865] File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 1418, in __getitem__
[2020-03-11 08:15:05,865] return self._getitem_tuple(key)
[2020-03-11 08:15:05,865] File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 2092, in _getitem_tuple
[2020-03-11 08:15:05,865] self._has_valid_tuple(tup)
[2020-03-11 08:15:05,865] File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 235, in _has_valid_tuple
[2020-03-11 08:15:05,865] self._validate_key(k, i)
[2020-03-11 08:15:05,865] File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 2031, in _validate_key
[2020-03-11 08:15:05,865] raise IndexError("positional indexers are out-of-bounds")
[2020-03-11 08:15:05,866] IndexError: positional indexers are out-of-bounds
[2020-03-11 08:15:05,866]
[2020-03-11 08:15:05,866] The above exception was the direct cause of the following exception:
[2020-03-11 08:15:05,866]
[2020-03-11 08:15:05,866] Traceback (most recent call last):
[2020-03-11 08:15:05,866] File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[2020-03-11 08:15:05,866] "__main__", mod_spec)
[2020-03-11 08:15:05,866] File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
[2020-03-11 08:15:05,867] exec(code, run_globals)
[2020-03-11 08:15:05,867] File "/src/d3m/d3m/__main__.py", line 6, in <module>
[2020-03-11 08:15:05,867] cli.main(sys.argv)
[2020-03-11 08:15:05,867] File "/src/d3m/d3m/cli.py", line 1111, in main
[2020-03-11 08:15:05,867] handler(arguments, parser)
[2020-03-11 08:15:05,867] File "/src/d3m/d3m/cli.py", line 1026, in handler
[2020-03-11 08:15:05,867] problem_resolver=problem_resolver,
[2020-03-11 08:15:05,867] File "/src/d3m/d3m/cli.py", line 516, in runtime_handler
[2020-03-11 08:15:05,868] problem_resolver=problem_resolver,
[2020-03-11 08:15:05,868] File "/src/d3m/d3m/runtime.py", line 2452, in fit_score_handler
[2020-03-11 08:15:05,868] fit_result.check_success()
[2020-03-11 08:15:05,868] File "/src/d3m/d3m/runtime.py", line 68, in check_success
[2020-03-11 08:15:05,868] raise self.error
[2020-03-11 08:15:05,868] File "/src/d3m/d3m/runtime.py", line 972, in _run
[2020-03-11 08:15:05,868] self._do_run()
[2020-03-11 08:15:05,868] File "/src/d3m/d3m/runtime.py", line 958, in _do_run
[2020-03-11 08:15:05,869] self._do_run_step(step)
[2020-03-11 08:15:05,869] File "/src/d3m/d3m/runtime.py", line 950, in _do_run_step
[2020-03-11 08:15:05,869] ) from error
[2020-03-11 08:15:05,869] d3m.exceptions.StepFailedError: Step 3 for pipeline fc1eee7f-6435-4001-9cf6-6d24330d9b1c failed.
primitives/v2020.1.9/ISI/d3m.primitives.feature_construction.echo_ib.DSBOX/1.0.0/pipeline_runs/26_radon_seed_pipeline_run.yml.gz
seems to be empty.
primitives/v2020.1.9/ISI/d3m.primitives.data_preprocessing.do_nothing.DSBOX/1.5.3/pipeline_runs/pipeline_run_38_sick_MIN_METADATA_2.yaml.gz
Seems to fail in scoring. Also has the following error:
[2020-03-11 08:20:37,400] WARNING:d3m.runtime:Pipeline run contains values for additional hyper-parameters: ['match_logic', 'negate']
[2020-03-11 08:20:37,401] WARNING:d3m.runtime:Pipeline run contains values for additional hyper-parameters: ['n_limit', 'return_result', 'use_semantic_types']
[2020-03-11 08:20:37,402] WARNING:d3m.runtime:Pipeline run contains values for additional hyper-parameters: ['drop_non_numeric_columns']
[2020-03-11 08:20:37,402] WARNING:d3m.runtime:Pipeline run contains values for additional hyper-parameters: ['add_index_columns', 'return_result', 'use_semantic_types', 'verbose']
[2020-03-11 08:20:37,403] WARNING:d3m.runtime:Pipeline run contains values for additional hyper-parameters: ['quantile_range_lowerbound', 'quantile_range_upperbound', 'with_centering', 'with_scaling']
[2020-03-11 08:20:37,403] WARNING:d3m.runtime:Pipeline run contains values for additional hyper-parameters: ['diagonal']