Sweep dimensions not correctly labelled
Issue was initially opened by Julian Weninger
Some recent change has changed the way sweep parameter are labelled in the DAG transformation tool.
Summary
I use a sweep of parameter lambda within list (and beta in a dict). However, the DAG-data has dimensions env_state_funcs.0.noise.atoh1.lambda and beta.
Steps to reproduce
My run config
parameter_space:
num_steps: 100
NotchDelta:
beta: !sweep
default: 8e-3
linspace: [0.01, 0.06, 4]
Environment:
env_state_funcs:
- noise:
atoh1:
mode: add
distribution: exponential
lambda: !sweep
default: 80
values: [40, 60]On the other hand this run config would produce the expected dimensions
parameter_space:
num_steps: 100
NotchDelta:
beta: !sweep
default: 8e-3
linspace: [0.01, 0.06, 4]
Environment:
lambda: !sweep
default: 80
values: [40, 60]
env_state_funcs:
- noise:
atoh1:
mode: add
distribution: exponential
lambda: !coupled-sweep
target_name: lambdaIn my plots cfg I do the following:
---
percolation_diagram: !pspace
based_on: .dag.generic.facet_grid
module: model_plots.NotchDelta
plot_func: percolation_diagram # calls facet_grid after modifying data
creator: multiverse
select_and_combine:
fields:
_cells: Cell_type
base_path: data/NotchDelta/CM
map_dim: beta
transform:
- .isel:
args: [!dag_tag _cells]
kwargs:
time: -1
- .sel:
args: [!dag_prev ]
kwargs:
lambda: !sweep
default: 80
values: [40, 60]
tag: cellsAs far as I can see it was successful in the pipeline on Jun 16th 11am and failed the first time June 19th (Friday) 2:40 p.m.
What is the current bug behaviour?
(More about the behaviour of the bug) It raises the following error -- dimension I want to select does not exist:
Traceback (most recent call last):
File "/home/julian/Utopia/dantro/dantro/utils/data_ops.py", line 1117, in apply_operation
return op(*op_args, **op_kwargs)
File "/home/julian/Utopia/dantro/dantro/utils/data_ops.py", line 989, in <lambda>
'.sel': lambda d, *a, **k: d.sel(*a, **k),
File "/home/julian/Utopia/utopia/build/utopia-env/lib/python3.8/site-packages/xarray/core/dataarray.py", line 1056, in sel
ds = self._to_temp_dataset().sel(
File "/home/julian/Utopia/utopia/build/utopia-env/lib/python3.8/site-packages/xarray/core/dataset.py", line 2065, in sel
pos_indexers, new_indexes = remap_label_indexers(
File "/home/julian/Utopia/utopia/build/utopia-env/lib/python3.8/site-packages/xarray/core/coordinates.py", line 396, in remap_label_indexers
pos_indexers, new_indexes = indexing.remap_label_indexers(
File "/home/julian/Utopia/utopia/build/utopia-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 254, in remap_label_indexers
dim_indexers = get_dim_indexers(data_obj, indexers)
File "/home/julian/Utopia/utopia/build/utopia-env/lib/python3.8/site-packages/xarray/core/indexing.py", line 220, in get_dim_indexers
raise ValueError(f"dimensions or multi-index levels {invalid!r} do not exist")
ValueError: dimensions or multi-index levels ['lambda'] do not exist
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/julian/Utopia/dantro/dantro/plot_mngr.py", line 603, in _invoke_creator
rv = pc(out_path=out_path, **plot_cfg)
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_base.py", line 223, in __call__
return self.plot(out_path=out_path, **cfg)
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_ext.py", line 232, in plot
self._plot_with_helper(out_path=out_path, plot_func=plot_func,
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_ext.py", line 390, in _plot_with_helper
args, kwargs = self._prepare_plot_func_args(plot_func,
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_psp.py", line 115, in _prepare_plot_func_args
return super()._prepare_plot_func_args(*args, **kwargs)
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_ext.py", line 532, in _prepare_plot_func_args
using_dag, kwargs = self._perform_data_selection(use_dag=use_dag,
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_base.py", line 338, in _perform_data_selection
dag_results = self._compute_dag(dag, **dag_params['compute'])
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_ext.py", line 650, in _compute_dag
return super()._compute_dag(dag, compute_only=compute_only,
File "/home/julian/Utopia/dantro/dantro/plot_creators/pcr_base.py", line 394, in _compute_dag
return dag.compute(**compute_kwargs)
File "/home/julian/Utopia/dantro/dantro/dag.py", line 1110, in compute
res = trf.compute()
File "/home/julian/Utopia/dantro/dantro/dag.py", line 402, in compute
res = self._perform_operation(args=args, kwargs=kwargs)
File "/home/julian/Utopia/dantro/dantro/dag.py", line 414, in _perform_operation
res = apply_operation(self._operation, *args, **kwargs)
File "/home/julian/Utopia/dantro/dantro/utils/data_ops.py", line 1120, in apply_operation
raise RuntimeError(
RuntimeError: Failed applying operation '.sel'! Got a ValueError: dimensions or multi-index levels ['lambda'] do not exist
args: (<xarray.DataArray 'Cluster_id' (env_state_funcs.0.noise.atoh1.lambda: 2, beta: 4, x: 8, y: 8)> What is the expected correct behaviour?
(Which behaviour would you have expected?)
Have dimensions lambda and beta within DAG framework
Relevant logs, screenshots, files...?
For further details see pipelines of vertex-model!29
Expand to see logs
Ideas how to fix this?
(Add them here, if you have any.)
Edited by Yunus Sevinchan