[Data transfer] Not possibe to upload data after the DSD is re-uploaded with a different structure - done through #65
When user follows the following scenario, the data upload fails:
- Create a DSD and DF and upload data there
- Delete DF and DSD
- Reupload DSD with a different structure (e.g change codelist, add/remove dimension etc) but with the same ID, agency and version and DF for it
- Try to upload data to the DF. The data upload will fail:
2/20/2019 2:06:48 PM ERROR Change has been detected on DSD SPC:SDG_DSD_SPC(1.0) since the last dataload: Code(s) removed from codelist IMF:CL_AREA(1.9) of dimension SPC:SDG_DSD_SPC(1.0).REF_AREA : _X, _Z, 1A, 1B, 1C, 1D, 1E, 1F, 1G, 1H, 1J, 1K, 1L, 1M, 1N, 1O, 1P, 1Q, 1R, 1S, 1T, 1U, 1U1, 1V, 1W, 4A, 4AA, 4B, 4C, 4D, 4DJ, 4E, 4F, 4G, 4H, 4I, 4J, 4J1, 4J2, 4J3, 4J4, 4J5, 4J6, 4J7, 4J8, 4J81, 4J810, 4J811, 4J812, 4J813, 4J814, 4J815, 4J816, 4J817, 4J818, 4J819, 4J82, 4J820, 4J821, 4J822, 4J823, 4J824, 4J825, 4J826, 4J827, 4J828, 4J829, 4J83, 4J830, 4J831, 4J832, 4J833, 4J834, 4J835, 4J836, 4J837, 4J838, 4J839, 4J84, 4J840, 4J841, 4J842, 4J85, 4J86, 4J87, 4J88, 4J89, 4M, 4R, 4S, 4T, 4T1, 4T2, 4T3, 4U, 4V, 4X, 4Y, 5A, 5AA, 5B, 5C, 5D, 5E, 5F, 5G, 5H, 5I, 5J, 5K, 5L, 5M, 5N, 5O, 5P, 5Q, 5R, 5S, 5T, 5U, 5W, 5X, 5Z, 5Z1, 5Z10, 5Z11, 5Z12, 5Z13, 5Z14, 5Z15, 5Z16, 5Z17, 5Z18, 5Z19, 5Z2, 5Z20, 5Z21, 5Z22, 5Z23, 5Z24, 5Z25, 5Z26, 5Z27, 5Z28, 5Z29, 5Z30, 5Z31, 5Z33, 5Z34, 5Z35, 5Z36, 5Z37, 5Z38, 5Z39, 5Z4, 5Z40, 5Z41, 5Z42, 5Z5, 5Z6, 5Z7, 5Z8, 5Z9, 6A, 6A1, 6A2, 6A3, 6A4, 6A5, 6A6, 6A7, 6A8, 6B, 6C, 6D, 6E, 6F, 6G, 6H, 6I, 6J, 6K, 6L, 6M, 6N, 6O, 6P, 6Q, 6Z, 7A, 7B, 7C, 7D, 7E, 7F, 7G, 7H, 7I, 7J, 7K, 7L, 7M, 7N, 7O, 9A, 9B, A1, A10, A109, A19, A1X, A1X0, A2, A29, A2A3, A2A39, A2X, A3, A39, A3X, A4, A41, A42, A49, A5, A5X, A6, A69, A7, A79, A7E, A7E9, A7X, A7X0, A8, A84, A9, A99, AE1, AE2, AQ, B0, B00, B09, B1, B10, B2, B20, B3, B30, B4, B40, B5, B50, B6, B60, C0, C1, C2, C3, C4, C5, C98, C99, CS9, CSH, CSH9, D0, D09, D1, D2, D3, D4, D5, D5A, D5B, D5C, DE1, DE2, E1, E16O, E19, E1E, E1E9, E1X, E1X0, E1X1, E2, E29, E2X, E3, E31, E39, E3X, E4, E49, E4X, E5, E59, E5X, E7, F1, F19, F1XM, F2, F3, F39, F4, F41, F42, F49, F5, F59, F6, F69, F7, F79, F8, F89, F97, F979, F98, F99, FR19, G0, G1, G2, G3, G4, G9, GB1, GB2, GB3, I1, I19, I2, I29, I3, I39, I4, I49, I5, I59, I6, I69, I7, I79, I8, I89, J1, J19, J2, J29, J3, J39, J4, J49, J5, J59, J6, J69, J7, J79, J8, J89, K1, K2, K3, K4, K5, K6, K7, K8, K8A, K9, K9A, L1, L2, L3, L4, L5, L6, L7, L8, M1, M2, M3, M4, M5, M6, M7, M8, N1, N2, O1, O19, O1P, O1X0, O2, O29, O3, O39, O3A, O3M, O3Z, O5, O6, O69, O6X0, O7, O79, O8, O89, O9, O99, P0, P09, P1, P10, P2, P3, P4, P8, P9, Q1, Q2, Q3, Q4, Q5, Q6, Q7, R1, R109, R11, R12, R121, R13, R14, R14M, R15, R16, R17, R18, R19, R2, R22, R220, R221, R222, R23, R24, R25, R251, R252, R253, R254, R255, R3, R4, R45, R5, R51, R6, R61, R62, R63, R7, R8, R9, S1, S19, S198, S1X, S1X0, S1X2, S1X3, S2, S29, S2X, S3, S35, S359, S35E, S35E9, S37, S379, S37E, S37E9, S37X, S37Z, S39, S3X, S3X0, S4, S49, S5, S59, S6, S6X, S7, S79, S8, S89, SUH, SUH9, TZXZ, TZZN, U0, U2, U20, U29, U3, U30, U39, U4, U49, U5, U7, U8, U9, UA, UG, UM, US, US1, US2, W0, W00, W08, W09, W10, W12, W190, W1X1, W1X2, X0, X1F, X1NF, X88, X999, XA6, XA69, XA7, XA79, XE2, XE3, XE39, XR21, XR23, XR24, XR25, XR29, XR30, XR32, XR40, XR41, XR42, XR43, XR44, XR55, XR99, XRC1, XS2, XS25, XS3, XS4, YUC, YUC9, Z1, Z10, Z11, Z12, Z3, Z4, Z5, Z6, Z8, Z9, P00, 5Z3, D6, K10, R52 2/20/2019 2:06:49 PM ERROR Exception of type 'DotStat.Transfer.Exception.TransferFailedException' was thrown.
I did the following test and everything worked properly for me:
1.- Create a DSD and a DF with no contentconstraint 2.- Upload data to it 3.- Delete the DSD 4.- Recreate the DSD with one less dimension 5.- Create the DF with no contenconstraint 6.- Upload data
As a workaround, I managed to upload data with no errors by deleteing all artefacts (DF, DSD, CS, CLs) and recreate all with the changes, and only then I was able to upload data.
What I did:
- Deleted all the structures from DLM (DSD, DF,Codelists, Concept schemes)
- Checked the Artefact table and the artefacts were still there (IDs 2-16 for the first data upload and ID 1004 CL_AREA for the second David’s try)
- I uploaded the DSD and the DF
- I tried to upload the data and it failed with the same error.
- I updated the version 3 using the query update dbo.Artefact set VERSION_3=99999 where ART_ID<20;
- I tried to upload the data. It failed with a different error: Unknown member for [EDUCATION_LEV:ISCED11_1], at row [33]
- I fixed the education code in the Excel file
- I tried to upload the data once again. It failed with a different error: Change has been detected on DSD SPC:SDG_DSD_SPC(1.0) since the last dataload: Code list of dimension SPC:SDG_DSD_SPC(1.0).FREQ changed to SDMX:CL_FREQ(2.0).
- I checked the Artefact table and all the artefacts got replicated during the first data upload (that was not successful) – IDs 1005-1020. I am not sure if this should be done during the unsuccessful data upload or not, so we might have two issues here: i. Triggering creation of artefacts in Artefact table ii. Deleting codelist artefacts when the codelist is deleted
As a workaround the following was done:
- Delete content of ARTEFACT and COMPONENT tables
- Deletion of CL_{id} tables and related availability tables (with the exception of CL_TIME)
Zsolt kept the copies of ARTEFACT and COMPONENT tables for the further analysis. For some reason it seems that FREQ and SERIES dimensions of the DSD got mapped to the old versions. Based on the IDs it seems that there were a few attempts already that failed at the SERIES or REF_AREA dimensions. And based on the tables it seems that the DSD row was not deleted when the update to 99999 happened.