Simplify the logic of mappingset creation to always regenerate mappingsets
The logic to regenerate mappingsets is currently done in a way that it updates/recreates the mappingsets only when there have been changes that require it; For example new PIT, update of PIT release date. The reason was that the EUROSTAT libraries that update the mappingsets were very slow (10% of the import process). This logic is spread across multiple places of the code and often leads to errors in the functionality, when the mappingsets must be updated, but they were not.
The logic in the EUROSTAT libaries has been refactored to be more performance, therefore the transfer service can update in every import the mappingsets, at a minimal performance cost. This will remove complexity in the code and prevent bug tickets related to this.
-
Always regenerate mappingsets in for the functions, for all related dataflows: - init/dataflow
- import/sdmxFile
- validate/sdmxFile
- pointintime/rollback
- pointintime/restoration
- transfer/dataflow
- validate/transferDataflow
- transfer/dsd
- validate/transferDsd
Steps to recreate 4 related bugs:
-
Case 1 - Import referential metadata using a dataflow reference
- Import referential metadata to create a PIT, using a DSD reference.
- Check that the PIT mappingsets for all dataflows belonging to this DSD, are updated with the PIT information.
-
Case 2 - Import referential metadata to create a PIT, using a DSD reference.
- Import referential metadata using a dataflow reference.
- Check that the PIT mappingsets for all dataflows belonging to this DSD are updated with the PIT information.
- Check that any new referential metadata imports, using a dataflow reference now target the existing PIT version.
-
Case 3 - Import referential metadata to create a PIT using a dataflow reference - Make sure that this is the first import ever any dataflow (using dataflow reference) belonging to the same DSD and also the fist import ever for this DSD (Using DSD reference).
- Import a new PIT that leads to an update of the release date, regardless of which artefact you use as reference.
- Check that the mappingsets are updated with the new PIT release date.
-
Extra bug to be fixed: -
upload one DSD + 2 related DFs OECD.SDD.SDPS-DF_TEST_REF_METADATA_1-DF_TEST_REF_METADATA_2-1.0.xml
-
upload data for dataflow_1 OECD.SDD.SDPS-DF_TEST_REF_METADATA_1-1.0-data.csv
-
upload data for dataflow_2 OECD.SDD.SDPS-DF_TEST_REF_METADATA_2-1.0-data.csv
As a result, an Actual Content Constraint must be created for dataflow_1 and for dataflow_2.