Data is not transferred from one space to another even if the transfer service says it was "successful"
1) Main issue to be solved
I am trying to transfer data uploaded to OECD.GOV:DF_GOV_CORE_RESULTS(2.0) from OECD-design to OECD-staging.
The underlying structures are NOT EXACTLY the same (see details below), but their Allowed Content Constraints are the same. Both dataflows have just been re-initiated. Before the transfer, the source dataflow has 902 observations, the target dataflow has 0 observations. MSDs are not linked.
The data transfer request registration was successful.
When I am checking the e-mail, it looks like everything is successful.
However, when checking the target dataflow, it's Actual Content Constraints still shows 0
in the sdmx_metrics
-obs_count
annotation and the NSI still returns NoRecordsFound
: https://nsi-demo-oecd-staging.siscc.org/rest/data/OECD.GOV,DF_GOV_CORE_RESULTS,2.0/.......?dimensionAtObservation=AllDimensions
S=Thus, the data actually was not transferred.
Using transfer with advanced validations has the same erroneous result.
Details to structural differences:
- The CategorySchemes are different (but this doesn't matter)
- Some codelists differ in versions, code descriptions or locals (but this doesn't matter)
- The order of dimensions are different (and this might indeed matter):
OECD-design (source) | OECD-staging (target) |
---|---|
FREQ | FREQ |
REF_AREA | MEASURE |
MEASURE | REF_AREA |
UNIT_MEASURE | AGE |
AGE | SEX |
SEX | EDUCATION_LEV |
EDUCATION_LEV | CATEGORY |
CATEGORY | UNIT_MEASURE |
TIME_PERIOD | TIME_PERIOD |
This behaviour could be replicated in qa-stable (source) and qa-reset (target). The source structures and data and the target structures are attached here below.
2) Related side-issue to be solved
Without the Allowed CC in the target space, when transferring the data from the source to the target then the transfer fails with the non-comprehensive error message:
ERROR The processing of observations was stopped and dropped due to one or more errors. Please read the details of the error(s) to see how to resolve it/them.
FATAL Index was outside the bounds of the array.
A subsequent dataflow (re-)init also results in the message "Index was outside the bounds of the array."
Expected outcome
The preferred solution for both issues would be that the transfer accepts and correctly deals with differences in the dimension orders as long as all DSD components have the equivalent representations (items/types).
Only if this is not possible, then the transfer should fail with an appropriate (informative) error message and must not corrupt the Data DB structures.
Example files
- source structures: OECD.GOV-DF_GOV_CORE_RESULTS-2.0-all.xml
- source data: OECD.GOV-DF_GOV_CORE_RESULTS-2.0-data.csv
- target structures with CCs for issue 1: OECD.GOV-DF_GOV_CORE_RESULTS-2.0-all_target.xml
- target structures without CCs for issue 2: OECD.GOV-DF_GOV_CORE_RESULTS-2.0-Target-Structures-Without-CCs.xml