3.0.1 Improve error messages for mistakes in the csv layout - SDMXRI-1494
When uploading a csv file that has missing or additional columns or column headers for dataflow, dimensions, measure ("OBS_VALUE") or attributes, then all layout issues should be mentioned: all expected and all missing columns and column headers should be listed.
Today, only one issue is given, and the user needs to recursively correct the csv input and retry until all issues are detected, notified and corrected.
Example: OECD:AIR_EMISSIONS_DF(1.0) OECD_AIR_EMISSIONS_DF_1.0_structure.xml
Examples of incomplete, insufficient or wrong error messages to be corrected:
-
no dimensions, measure or attributes: Carriage return is not a valid delimiter.
OECD_AIR_EMISSIONS_DF_1.0_missing_dimensions_and_measure.csv
To be corrected to this message:
Error: In line 1, column header row, column header missing for dimension COU. Add the COU dimension column or correct the corresponding column header to COU and resubmit the request.
Error: In line 1, column header row, column header missing for dimension POL. Add the POL dimension column or correct the corresponding column header to POL and resubmit the request.
Error: In line 1, column header row, column header missing for dimension VAR. Add the VAR dimension column or correct the corresponding column header to VAR and resubmit the request.
Error: In line 1, column header row, column header missing for dimension TIME_PERIOD. Add the TIME_PERIOD dimension column or correct the corresponding column header to TIME_PERIOD and resubmit the request.
Error: In line 1, column header row, column header missing for measure OBS_VALUE. Add the OBS_VALUE measure column or correct the corresponding column header to OBS_VALUE and resubmit the request. -
all dimensions missing except one; measure and attributes missing: Dimension with ID [POL] is missing in the csv file
OECD_AIR_EMISSIONS_DF_1.0_missing_dimensions_and_measure2.csv
To be corrected to this message:
Error: In line 1, column header row, column header missing for dimension POL. Add the POL dimension column or correct the corresponding column header to POL and resubmit the request.
Error: In line 1, column header row, column header missing for dimension VAR. Add the VAR dimension column or correct the corresponding column header to VAR and resubmit the request.
Error: In line 1, column header row, column header missing for dimension TIME_PERIOD. Add the TIME_PERIOD dimension column or correct corresponding the column header to TIME_PERIOD and resubmit the request.
Error: In line 1, column header row, column header missing for measure OBS_VALUE. Add the OBS_VALUE measure column or correct the corresponding column header to OBS_VALUE and resubmit the request. -
no measure, no attributes: Attribute with ID [OBS_STATUS] is missing in the csv file
OECD_AIR_EMISSIONS_DF_1.0_missing_measure3.csv
To be corrected to this message:
Error: In line 1, column header row, column header missing for measure OBS_VALUE. Add the OBS_VALUE measure column or correct the corresponding column header to OBS_VALUE and resubmit the request. -
no attributes: Attribute with ID [OBS_STATUS] is missing in the csv file
OECD_AIR_EMISSIONS_DF_1.0_missing_attributes4.csv
To be corrected in #61 (closed) to NOT through an error because none of the attributes are mandatory and those columns are allowed to be omitted.
To be corrected: In case attributes are mandatory then the error message should be:
Error: In line 1, column header row, column header missing for mandatory attribute UNIT. Add the UNIT attribute column or correct the corresponding column header to UNIT and resubmit the request. -
no measure: Index was outside the bounds of the array.
OECD_AIR_EMISSIONS_DF_1.0_missing_measure5.csv
To be corrected to this message:
Error: In line 1, column header row, column header missing for measure OBS_VALUE. Add the OBS_VALUE measure column or correct the corresponding column header to OBS_VALUE and resubmit the request. -
wrong measure (anything else than "OBS_VALUE"): Index was outside the bounds of the array.
OECD_AIR_EMISSIONS_DF_1.0_wrong_measure6.csv
To be corrected to this message:
Error: In line 1, column header row, column header missing for measure OBS_VALUE. Add the OBS_VALUE measure column or correct the corresponding column header to OBS_VALUE and resubmit the request. -
additional columns: 4 observations were processed | No observations were processed | Dataflow [OECD:AIR_EMISSIONS_DF(1.2)] not found in [SIS-CC-reset] dataspace.
OECD_AIR_EMISSIONS_DF_1.0_additional_column7.csv
To be corrected in #61 (closed) to NOT through an error because additional columns are allowed. -
dimensions not in DSD order: 4 observations were processed | No observations were processed | Dataflow [OECD:AIR_EMISSIONS_DF(1.2)] not found in [SIS-CC-reset] dataspace.
OECD_AIR_EMISSIONS_DF_1.0_diff_dim_order8.csv
To be corrected in #61 (closed) to NOT through an error because dimension columns do not have to be provided in a specific order. -
attributes not in DSD order: 4 observations were processed | No observations were processed | Dataflow [OECD:AIR_EMISSIONS_DF(1.2)] not found in [SIS-CC-reset] dataspace.
OECD_AIR_EMISSIONS_DF_1.0_diff_attr_order9.csv
To be corrected in #61 (closed) to NOT through an error because attribute columns do not have to be provided in a specific order.
Note: Some of the current CSV load behaviour is incorrect and doesn't respect the SDMX-CSV standard:
-
Optional attributes are NOT required and imports without optional attributes must be accepted.
-
Additional columns (others than dataflow, dimensions, measure and attributes) are allowed and imports with such additional columns must be accepted. Those columns should simply be ignored.
-
The order of columns is free, except for the dataflow column that is always the first column. Imports with arbitrary column orders, where at least the dataflow column is placed first, must be accepted.
A separate ticket is created for these issues, see: #61 (closed).
Error messages are sometimes too technical, and need clarification. A separate ticket is created for this issue, see: #53 (closed).