Data upload uses wrong dataflow version number
When uploading a csv file that uses dataflow OECD:DF_AIR_EMISSIONS(1.1) and a transfer service call with parameter OECD:DF_AIR_EMISSIONS(1.1) using:
curl -X POST "http://transfer-qa-oecd.redpelicans.com/1.2/import/sdmxFile" -H "accept: application/json" -H "authorization: Bearer XXXXX" -H "Content-Type: multipart/form-data" -F "dataspace=reset" -F "dataflow=OECD:DF_AIR_EMISSIONS(1.1)" -F "file=@OECD,DF_AIR_EMISSIONS,1.1 small.csv;type=application/vnd.ms-excel"
I wrongly receive the error:
Date Level Message
01/28/2020 16:25:44 NOTICE Successfully registered request with ID: 92
01/28/2020 16:25:44 NOTICE 1568 observations were processed Details
01/28/2020 16:25:44 WARN No observations were processed Details
01/28/2020 16:25:44 ERROR Dataflow [OECD:DF_AIR_EMISSIONS(1.2)] not found in [reset] dataspace. Details
The structures and data used are attached.
To reproduce:
-
Load structure: OECD-DF_AIR_EMISSIONS-1.1-all.xml
or from \dotstatcor-dev1\Temp\Uploads\OECD-DF_AIR_EMISSIONS-1.1-all.xml -
Try to load data: OECD_DF_AIR_EMISSIONS_1.1_small.csv
or from \dotstatcor-dev1\Temp\Uploads\OECD,DF_AIR_EMISSIONS,1.1 small.csv
From @aleksei-oecd :
i think with probability 90% i found the place where error happens. regarding the final message
it's in the UpdateCodeListAvailabilityInfo method
and probably here more then 1 bug. Teroretically as exception happens in the transfer method the rollback should clean everything that was written
https://gitlab.com/sis-cc/.stat-suite/dotstatsuite-core-transfer/blob/develop/DotStat.Transfer/Consumer/SqlConsumer.cs
error happens when executing line 301, and it should be later rollbacked with line 306
so for the first bug "[OECD:DF_AIR_EMISSIONS(1.2)] not found" in our database there is a record in ARTEFACT for version 1.2 (UpdateCodeListAvailabilityInfo looks for all dataflows that has the same DSD as currently uploaded dataflow)
but in eurostat's database there is no (deleted ?) dataflow with version 1.2
again issue related to un-sycronized nature of data-db && structure-db
btw UpdateCodeListAvailabilityInfo is not optimal in using resources. But this is again another story