Hub feature: Allow using externally defined dataflows
In the DE search indexer, consider an SDMX data source that contains the categoryscheme and categorised dataflows, which are to be indexed, but that some dataflows are stored only as stubs (without content, e.g. without the link to its DSD). The full definition and content of these dataflows are actually stored externally. Therefore the locally stored dataflow stubs include the references (URL link) to the original external full dataflow definition with the following artefact properties: isExternalReference=true, and link{external structure link}. In such as case, the DE search indexer must use the given external URL to the dataflow rather then self-constructing the URL for the dataflow structure query from the local SDMX data space, in order to obtain the full dataflow information.
Example of dataflow stub with external reference retrieved through http://sdmx-ws-root/categoryscheme/[agencyID]/[categorySchemeID]/[version]/?references=dataflow:
{
"meta": {
# meta object #
},
"data": {
"dataflows": [
{
"id": "DF_SDG_ALL_SDG_A871_SEX_AGE_RT",
"links": [
{
"rel": "self",
"type": "dataflow"
},
{
"href": "https://ilo.org/sdmxtest/rest/dataflow/ILO/DF_SDG_ALL_SDG_A871_SEX_AGE_RT/1.0",
"rel": "external"
}
],
"version": "1.0",
"agencyID": "ILO",
"isExternalReference": true,
"isFinal": true,
"name": "SDG indicator 8.7.1 - Proportion of children engaged in economic activity",
"names": {
"en": "SDG indicator 8.7.1 - Proportion of children engaged in economic activity"
}
}
]
}
}
In this example, the following query needs to be made to retrieve the full dataflow information: https://ilo.org/sdmxtest/rest/dataflow/ILO/DF_SDG_ALL_SDG_A871_SEX_AGE_RT/1.0/?references=all
Objective: change the structure retrieval algorithm of the search indexer to not solely rely on pre-configured internal data spaces but to allow also indexing externally hosted dataflows can still be indexed and
Note:
The linked ticket dotstatsuite-data-explorer#96 (moved) describes the required changes in the DE data view page (as well as the Viewer application) to also handle the case of such externally defined dataflows.