3.1 Additional import validations for time dimension
NOTE: This should be implemented together with dotstatsuite-core-common#113 (closed).
As Patrice,
I want the upload system to check the imported data in regard to the respect of the defined time dimension representation (e.g. business data or daily data)
So that obviously invalid data cannot be loaded.
Specification:
Extension of the validation with additional basic validation in the data import methods
- the time dimension values must meet the representation requirements
When the allowed representation of the measure is restricted, e.g. to "GregorianDay" (YYYY-MM-DD), then this restriction needs to be applied when loading data. Today it is possible to use "2009-01" as data format.
Structure example: DAILY_DSD.xml
<structure:TimeDimension id="TIME_PERIOD" position="16">
<structure:ConceptIdentity>
<Ref id="TIME_PERIOD" maintainableParentID="SDG_CONCEPTS" maintainableParentVersion="1.0" agencyID="IAEG-SDGs" package="conceptscheme" class="Concept" />
</structure:ConceptIdentity>
<structure:LocalRepresentation>
<structure:TextFormat textType="GregorianDay" />
</structure:LocalRepresentation>
</structure:TimeDimension>
Data example: UNSD-DF_JENS_DAILY-1.0-data__2_.csv (NOTE: The value for the UNIT_MEASURE attribute might need to be changed depending on the current common codelist content.)
DATAFLOW,FREQ,REPORTING_TYPE,SERIES,REF_AREA,SEX,AGE,URBANISATION,INCOME_WEALTH_QUANTILE,EDUCATION_LEV,OCCUPATION,CUST_BREAKDOWN,COMPOSITE_BREAKDOWN,DISABILITY_STATUS,ACTIVITY,PRODUCT,TIME_PERIOD,OBS_VALUE,OBS_STATUS,UNIT_MULT,UNIT_MEASURE,NATURE,COMMENT_OBS,TIME_COVERAGE,UPPER_BOUND,LOWER_BOUND,BASE_PER,TIME_DETAIL,SOURCE_DETAIL,COMMENT_TS,GEO_INFO_URL,GEO_INFO_TYPE,CUST_BREAKDOWN_LB,DATA_LAST_UPDATE
UNSD:DF_JENS_DAILY(1.0),D,N,SI_POV_NAHC,ET,_T,_T,_T,_T,_T,_T,_T,_T,_T,_T,_T,2020-02-02,45.5,A,0,NUMBER,_X,,,,,,1995,Source,,,,,
UNSD:DF_JENS_DAILY(1.0),D,N,SI_POV_NAHC,ET,_T,_T,_T,_T,_T,_T,_T,_T,_T,_T,_T,2009-01-01,44.2,A,0,NUMBER,_X,,,,,,2000,Source,,,,,
UNSD:DF_JENS_DAILY(1.0),D,N,SI_POV_NAHC,ET,_T,_T,_T,_T,_T,_T,_T,_T,_T,_T,_T,2009-01,44.2,A,0,NUMBER,_X,,,,,,2000,Source,,,,,
The monthly data point was loaded which should not be allowed:
https://de-qa.siscc.org/vis?lc=en&df[ds]=qa%3Astable&df[id]=DF_JENS_DAILY&df[ag]=UNSD&df[vs]=1.0&av=true&pd=%2C&dq=D..............