Full support of hierarchical codelists
As Anastassia
I want to use SDMX 2.1 Hierarchical Codelist (HCL) artefacts to change the hierarchy display of specific dimensions for specific dsds or dataflows on the DE visualisation page
So that dimension hierarchies are not limited to the simple hierarchy (one parent for each child only) defined in the underlying code lists, but allow for more flexibility and respecting the hierarchies defined in statistical classifications.
Specifications
- It concerns only the filter and table display on the DE visualisation page. In homepage and search facets the order and hierarchy of the original codelists remain to be used.
- Specific hierarchies to be applied are defined through a DSD and/or Dataflow annotation of type
HIER_CONTEXT
See: https://sdmx.org/wp-content/uploads/Guidelines-on-the-use-of-SDMX-Annotations.pdf, page 12.
This is a work-around for the not yet available SDMX 3.0 HierarchyAssociation artefact.
In case both are present, the Dataflow annotation takes precedence over the DSD annotation.
The (comma-separated) link(s) between the dimension and the hierarchy in the hierarchical codelist artefact is provided in the Annotation Title for non-localised hierarchies and in the Annotation Text for localised hierarchies in the following form:<component id>:<HCL agency>:<HCL id>(<HCL version>).<hierarchy id>,<component id>:<HCL agency>:<HCL id>(<HCL version>).<hierarchy id>,...
Example:REF_AREA:OECD:HCL_TEST(1.0).CONT_EN
provides the alternative hierarchy defined in artefact with urnurn:sdmx:org.sdmx.infomodel.codelist.Hierarchy=REF_AREA:OECD:HCL_TEST(1.0).CONT_EN
for theREF_AREA
dimension for the given dataflow or for all dataflows of the given DSD. - Therefore any whole dimension can be linked to a hierarchy, or any dimension locale can be linked to a hierarchy.
- These specific hierarchies overwrite the hierarchy and order of codes defined in the original code lists in a specific context (dsd+dataflow+dimension+language). For example, the original hierarchy of the "Reference area" codelist can be replaced by hierarchies better representing OECD/non-OECD countries, countries by continents, DAC countries by different groups, overseas territories displayed under their parent country, or in a flat list in an alphabetical order, etc.
- Since these hierarchies allow for more complex parent-client relationships than the original codelists do, especially one child can now have several parents (e.g France is part of World, of OECD countries, of EU countries and of EA countries etc), the filters and table views need to account for this. A code attached to several parents needs to be shown as child of each of the parents, except in table views where a parent might not be provided in the data message.
- Table view: The observations attached to the multi-parent codes are to be re-displayed (duplicated) for each present parent.
- Filter: When a code is a child of more than one parent, then it is displayed multiple times in the hierarchy. The selection approach could remain per individually displayed code instance or alternatively selecting any child once will also select it everywhere in the hierarchy (to be tested and challenged between JS & PM team during implementation), however the URL and the SDMX data query take the selected code IDs only once. Also, a child with multiple parents in the same filter should only be counted as one in the filter count green icon. When a visualisation page is constructed freshly from the URL (with unique code IDs), then the filter pre-selects all instances of the provided codes in the hierarchy.
-
Used filters: A selected child with multiple parents should appear only once, and the tooltip should display per line:
- once only the child if there is at least one child without selected parent
- the breadcrumb of each child where the parents are selected
(to be reviewed and agreed between JS & PM teams during implementation)
- The Excel download table view should include the multiple hierarchy representations as displayed in the table.
- NEW (12/04/2022): The codes to be displayed that are not listed in the HCL will still be displayed, but at root level and (if technically not too tricky) at the top of the tree. This will allow data owners to pick up the modelling mistake early on in the validation step and to add the missing codes to the HCL.
Examples
-
The here attached hierarchies "CONT_EN" and "CONT_FR" are linked in the corresponding dataflow or dsd annotation to the REF_AREA dimension:
-
Hierarchy "CONT_EN": first World then OECD, all others are not displayed because out of validity date
Hierarchy "CONT_FR": first OECD then World, all others are not displayed because out of validity date
HCL_-_Hierarchy_by_continents.xml -
Dataflows "DF_AREA_LOCALISED_HIERARCHY" (dataflow annotation) and "DF_AREA_DSD_LOCALISED_HIERARCHY" (dsd annotation) dimension REF_AREA
en: using hierarchy "CONT_EN": first World then OECD, all others are not displayed because out of validity date
fr: using hierarchy "CONT_FR": first OECD then World, all others are not displayed because out of validity date
Dataflows "DF_AREA_NONLOCALISED_HIERARCHY" (dataflow annotation) and "DF_AREA_DSD_NONLOCALISED_HIERARCHY" (dsd annotation) dimension REF_AREA
all languages: using hierarchy "CONT_EN": first World then OECD, all others are not displayed because out of validity date
Dataflow "DF_AREA_NO_HIERARCHY" - no special hierarchy
TEST-DF_AREA-1.0-all.xml -
Data files (1 for each dataflow)
TEST-DF_AREA_LOCALISED_HIERARCHY-1.0-data__1_.csv
TEST-DF_AREA_NONLOCALISED_HIERARCHY-1.0-data__1_.csv
TEST-DF_AREA_NO_HIERARCHY-1.0-data__1_.csv
TEST-DF_AREA_DSD_LOCALISED_HIERARCHY-1.0-data__1_.csv
TEST-DF_AREA_DSD_NONLOCALISED_HIERARCHY-1.0-data__1_.csv
SDMX-JSON snippet:
"data": {"hierarchicalCodelists": [{
"id": "HCL_TEST",
"links": [{
"rel": "self",
"urn": "urn:sdmx:org.sdmx.infomodel.codelist.HierarchicalCodelist=TEST:HCL_TEST(1.0)"
}],
"version": "1.0",
"agencyID": "TEST",
"isFinal": false,
"validFrom": "2022-01-01T00:00:00",
"validTo": "2030-12-31T00:00:00",
"name": "Reference area hierarchical Codelist",
"names": {"en": "Reference area hierarchical Codelist"},
"hierarchies": [
{
"id": "CONT_EN",
"links": [{
"rel": "self",
"urn": "urn:sdmx:org.sdmx.infomodel.codelist.Hierarchy=TEST:HCL_TEST(1.0).CONT_EN"
}],
"name": "Hierarchy by continents - World first",
"names": {"en": "Hierarchy by continents - World first"},
"leveled": false,
"hierarchicalCodes": [
{
"id": "W",
"links": [{
"rel": "self",
"urn": "urn:sdmx:org.sdmx.infomodel.codelist.HierarchicalCode=TEST:HCL_TEST(1.0).CONT_EN.W"
}],
"code": "urn:sdmx:org.sdmx.infomodel.codelist.Code=TEST:CL_AREA(1.0).W",
"codeID": "W",
"codelistAliasRef": "CL_AREA@TEST@10",
"hierarchicalCodes": [
{
"id": "A",
"links": [{
"rel": "self",
"urn": "urn:sdmx:org.sdmx.infomodel.codelist.HierarchicalCode=TEST:HCL_TEST(1.0).CONT_EN.W.A"
}],
"code": "urn:sdmx:org.sdmx.infomodel.codelist.Code=TEST:CL_AREA(1.0).A",
"codeID": "A",
"codelistAliasRef": "CL_AREA@TEST@10"
},
...
],
"includedCodelists": {"CL_AREA@TEST@10": "urn:sdmx:org.sdmx.infomodel.codelist.Codelist=TEST:CL_AREA(1.0)"}
}]},
Technical Note
There are 2 ways of retrieving the link from an hierarchical code to the code of the original codelist:
- get
hierarchicalCode
'scode
property with the urn (including codelist and code ID) - get codelist(s) from hierarchicalCodelist "includedCodelists" property with codelist alias and urn
gethierarchicalCode
'scodeID
(for code ID) andcodelistAliasRef
for codelist alias
Original specification based on SDMX 3.0 to be implemented later
Use SDMX 3.0 Hierarchy and HierarchyAssociation artefacts to dipslay different hierarchies and code orders in Dataflows on the visualisation page than in the original code list.
- On data visualisation page display hierarchies as defined in HierarchyAssociation(s) attached to a Dataflow or a DSD. For example, a different hierarchy may be displayed in the Reference area related-dimensions depending on the HierarchyAssociation(s) attached to a DF or DSD - OECD/non-OECD countries, countries by continents, DAC countries by different groups, overseas territories displayed under their parent country or in a flat list in an alphabetical order etc.
- Hierarchy and order of codes defined through Hierarchy and HierarchyAssociation override the hiearchy and codes in the original code lists.
- If one code has several parents (e.g France is part of OECD countries, EU countries, EA countries etc), then it should be displayed in the sections of all parents (the same observation may be displayed in different places of the table and filters, see the screenshot).
- These hierarchies should be applied to both, table filters and table visualisation.
- In homepage and search facets orders and hiearchies of the original code list are used.
- It is assumed that up to one hierarchy is defined per DSD component.
- If there are HierarchyAssociations defined for the same component on the DSD and DF level, then hierarchy attached to DF should be used.
Open questions:
- Check in the standard what is expected to be provided in the HierarchyAssociation ContextObject - is it Dimension or Code list? Does it cover the case when different hierachies are needed in the dimensions that are based on the same code list (e.g. Donor and Recipient)
- Would it be possible to define different orders of codes in hierarchy depending on language?
- Discussion started with ILO - which definition of orders takes presedence (in original code list or hierarchy)?
- It is assumed that one hierarchy is defined per DSD component. Should the first one be used if for some reason there are more found?
TODO:
-
Check if the above requirements are inline with ECO's usage of hiearchies -
Retrieve original requirements from ISTAT / validate these requirements with ISTAT