Skip to content

Handle properly the exception raised in case of no data for a sw index

deck requested to merge deckbsd/vinvelivaanilai:handle-no-data-to-contate-crash into master

This is the first part to fix the crash of Polaris if no data for a specific sw is found (see #169 on Polaris).

When such a case occur, an exception from panda occur :

polaris fetch \
	       --import_file `pwd`/./44878-482-20210426T003943Z-short.csv \
	       --ignore_errors \
	       --skip_normalizer \
	       Opssat1 \
	       ./opssat1-normalized_frames-short.json
       2021-04-30 13:41:29,714 - polaris.fetch.data_fetch_decoder - INFO - Satellite: id=44878 name=Opssat1 decoder=opssat1
       ...
       2021-04-30 13:41:30,535 - vinvelivaanilai.space_weather.sw_file_fetch - INFO - Fetching file at /tmp/2021Q1_DPD.txt
       2021-04-30 13:41:30,782 - vinvelivaanilai.space_weather.sw_file_fetch - INFO - 550 2021Q1_DPD.txt: No such file or directory
       2021-04-30 13:41:30,782 - vinvelivaanilai.space_weather.sw_file_fetch - INFO - Fetching file at /tmp/2021Q2_DPD.txt
       2021-04-30 13:41:31,031 - vinvelivaanilai.space_weather.sw_file_fetch - INFO - 550 2021Q2_DPD.txt: No such file or directory
       Traceback (most recent call last):
	 File "/home/aardvark/dev/src/polaris/.venv/bin/polaris", line 33, in <module>
	   sys.exit(load_entry_point('polaris-ml', 'console_scripts', 'polaris')())
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
	   return self.main(*args, **kwargs)
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
	   rv = self.invoke(ctx)
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
	   return _process_result(sub_ctx.command.invoke(sub_ctx))
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
	   return ctx.invoke(self.callback, **ctx.params)
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
	   return callback(*args, **kwargs)
	 File "/home/aardvark/dev/src/polaris/polaris/polaris.py", line 101, in cli_fetch
	   data_fetch_decode_normalize(
	 File "/home/aardvark/dev/src/polaris/polaris/fetch/data_fetch_decoder.py", line 206, in data_fetch_decode_normalize
	   sw_columns_names, preprocessed_sw = fetch_preprocessed_sw(
	 File "/home/aardvark/dev/src/polaris/polaris/fetch/fetch_import_sw.py", line 284, in fetch_preprocessed_sw
	   sw_data = fetch_or_import_sw(start_date, end_date, cache_dir, sat,
	 File "/home/aardvark/dev/src/polaris/polaris/fetch/fetch_import_sw.py", line 195, in fetch_or_import_sw
	   data = fetch_sw(local_start_date, local_end_date, cache_dir, indices)
	 File "/home/aardvark/dev/src/polaris/polaris/fetch/fetch_import_sw.py", line 68, in fetch_sw
	   temp_df = sw_file_fetch.fetch_indices(index, start_date, end_date,
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/vinvelivaanilai/space_weather/sw_file_fetch.py", line 87, in fetch_indices
	   data = _fetch_attempt(index, start_date, final_date, directory)
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/vinvelivaanilai/space_weather/sw_file_fetch.py", line 147, in _fetch_attempt
	   return sw_extractor.extract_data_from_multiple(index, file_name_list)
	 File "/home/aardvark/dev/src/polaris/.venv/lib/python3.8/site-packages/vinvelivaanilai/space_weather/sw_extractor.py", line 175, in extract_data_from_multiple
	   return pd.concat(dfs)
	 File "/home/aardvark/dev/src/polaris/.venv/lib64/python3.8/site-packages/pandas/core/reshape/concat.py", line 285, in concat
	   op = _Concatenator(
	 File "/home/aardvark/dev/src/polaris/.venv/lib64/python3.8/site-packages/pandas/core/reshape/concat.py", line 342, in __init__
	   raise ValueError("No objects to concatenate")
       ValueError: No objects to concatenate
       make: *** [Makefile:53: fetch-short] Error 1

The exception doesn't give proper explanations about what happen (from a "vinvelivaanilai" business point of view).

Result :

vinvelivaanilai now raise a NoSpaceWeatherForIndex exception when there are no data for an index with a text giving an explanation of what just happened.

Example if no DPD data where found for a giving date range.

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    fetch_indices("DPD", start_date, end_date)
  File "/home/deckbsd/repos/vinvelivaanilai/vinvelivaanilai/space_weather/sw_file_fetch.py", line 87, in fetch_indices
    data = _fetch_attempt(index, start_date, final_date, directory)
  File "/home/deckbsd/repos/vinvelivaanilai/vinvelivaanilai/space_weather/sw_file_fetch.py", line 147, in _fetch_attempt
    return sw_extractor.extract_data_from_multiple(index, file_name_list)
  File "/home/deckbsd/repos/vinvelivaanilai/vinvelivaanilai/space_weather/sw_extractor.py", line 182, in extract_data_from_multiple
    raise NoSpaceWeatherForIndex(
vinvelivaanilai.space_weather.sw_extractor.NoSpaceWeatherForIndex: No space weather data for DPD index

Second part :

For fixing the issue 169 the second part is to handle this exception properly on the Polaris side by logging the information and continue the sw fetching process. (I will do it once this MR will be merged and a new vv deployed)

Edited by deck

Merge request reports