Advanced data validation should not block data uploads because of invalid DB content
As Anastassia,
I want that the advanced validation feature returns only a warning about inconsistencies in already uploaded data instead of the current error resulting in an import failure,
So that it becomes possible to upload a valid data file with advanced validation, e.g. in order to correct the previously uploaded invalid content.
Example
When uploading:
- structures: OECD.SDD.NAD-NAMAIN_NAMAIN_SNA_TABLE1-1.0-all__1_.xml
- invalid data with basic validation (to be uploaded first): Missing_mandatory_attributes.csv
- more invalid data with basic validation (to be uploaded second): Missing_mandatory_attributes2.csv
- valid data with advanced validation without changing data (to be uploaded third): Valid_data_file_small.csv
- valid data with advanced validation with changing data (to be uploaded again last): Valid_data_file_small.csv
then the user receives this (messy) errors in steps 4 and 5:
-
Instead, the import actions 4 and 5 (for both advanced validation without and with changing data) should not fail and the error be replaced by a warning about incorrect already existing db data. -
The warning messages about incorrect already existing db data are to be corrected, especially as the current locations of the issues are wrong and misleading:
Errors received if step 3 (more invalid data with basic validation) is skipped:
Errors received if step 3 (more invalid data with basic validation) is not skipped:
Instead, the warning should say "... _are found in the database at coordinate: _"
Tasks
For the following 3 advance validations do the following:
- MandatoryDatasetAttributeWithNullValueInDatabase
- MandatoryAttributeWithNullValueInDatabase
- MultipleValuesForDimGroupAttributeInDatabase
-
Re-introduce them, since they will be temporarly removed by this ticket -
Return a warning when such validation is encounterd (Allow the import to complete) -
Technical implementation: -
The validation step should be done after the merge from the staging to the real tables -
The validation should be done against the real tables
-
-
Introduce Advance validation for delete operations .- Validate mandatory attributes
Edited by jean-baptiste nonin