Allow uploading data with long textual attribute values for dataflows in first uploads and subsequent uploads
A new transfer is now available in QA http://transfer-qa-oecd.redpelicans.com/swagger/index.html with the version "version": "2.5.8+684f3b0563" which is higher than v2.4+.
So after testing with DF OECD:SPS_ATT_TEST_TFWithRelationship(11.1) in the reset space and inserting long comments,higher than 150 characters, the data transfer was not processed and the following message sent.
Dear DLM user, your transfer has been processed in the system, please see below the details: Summary Request ID: 32 Destination data space: reset Dataflow: OECD:SPS_ATT_TEST_TFWithRelationship(11.1) User: firstname.lastname@example.org Request status: Failed Transaction messages | Date | Level | Message | |------------------- |------------- |------------------------------------------- | |11/28/2019 12:55:27 |NOTICE |Successfully registered request with ID: 32 | |11/28/2019 12:55:30 |WARN |No observations were processed Details | |11/28/2019 12:55:30 |ERROR |String or binary data would be truncated. | The statement has been terminated.
But such data load with long comments works with new dataflow.
The following approach was decided with the developer team:
- Configuration parameter is used as a default setting for MaxTextAttributeLength, its default value should be 150.
- A new DSD annotation MaxTextAttributeLength or MAXTEXTATTRIBUTELENGTH can be set be the user for a specific DSD
- When new data tables (first uploads) are created then the default MaxTextAttributeLength is used in case a DSD-specific MaxTextAttributeLength annotation was not provided, otherwise the DSD-specific MaxTextAttributeLength is used.
- Whenever a DSD-specific MaxTextAttributeLength is changed after a first data upload, the transfer service will use the new setting to update the existing data table at the next data upload.
- When there is no DSD-specific MaxTextAttributeLength set and the default MaxTextAttributeLength value is increased after a first data upload, the transfer service will use the new setting to update the existing data table at the next data upload.
- Within the scope if this request we don't support the case when the DSD-specific or the default MaxTextAttributeLength value is decreased, and as such some existing text attributes may be too long for the new MaxTextAttributeLength value. In such a case the new MaxTextAttributeLength is ignored, the structures of the related tables are not changed and a warning message is added to the transaction log.
- If the annotation MaxTextAttributeLength has invalid (non-integer) value, the transfer service ignores it and uses the default MaxTextAttributeLength value to determine the actions needed. Also a warning message is added to the transaction logs about the invalid MaxTextAttributeLength value.