Search facet items ordered by explicit annotation "ORDER"
As Edgardo,
I want to have explicit (through annotations) code/category orders in codelists/categoryschemes applied to the facet items,
So that the facet item order follows a statistical meaning (instead of current hit number) and eases the understanding of these items or the availability of items (especially in a larger list).
Acceptance Criteria
-
Facet items displayed on the DE home and search result pages are ordered according to the explicit order of the codes/categories (sdmx "ORDER" annotation of data type integer) and according to the current locale.
<structure:Code id="Y5-14"> <common:Annotations> <common:Annotation> <common:AnnotationTitle>55</common:AnnotationTitle> <common:AnnotationType>ORDER</common:AnnotationType> <common:AnnotationText xml:lang="en">55</common:AnnotationText> <common:AnnotationText xml:lang="es">80</common:AnnotationText> <common:AnnotationText xml:lang="fr">85</common:AnnotationText> </common:Annotation> </common:Annotations> <common:Name xml:lang="en">5 to 14</common:Name> <common:Name xml:lang="es">5 a 14</common:Name> <common:Name xml:lang="fr">5 à 14</common:Name> <structure:Parent> <Ref id="_T"/> </structure:Parent> </structure:Code>
Note: Use the localised order value when provided, otherwise use the non-localised order value when provided.
-
In case an explicit "ORDER" annotation is not provided for an item then an order value "0" is assumed for that item.
-
In case of hierarchical facets, hierarchy always prevails over the the explicit "ORDER" annotation, and explicit order will be applied in each node/level of the hierarchy independently from other nodes/levels.
-
In case any two or more facet items have the same ORDER value then these items will be ordered like today based on hits. Note that the same ORDER values for different facet items are likely when several codelists are merged into a same facet (when the concept name is the same). Logically, the facet items are always first ordered by ORDER and then by hits.
Example:
- Aaa, Explicit order=5, Hit: 14
- Bbb, Explicit order=4, Hit: 4
- Ccc, Explicit order=3, Hit: 9
- Ddd, Explicit order=3, Hit: 8
- Eee, Explicit order=2, Hit: 11
- Fff, Hit: 12
- Ggg, Explicit order=1, Hit: 10
- Hhh, Hit: 20
- Iii, Explicit order=1, Hit: 15
- Jjj, Hit: 4
Display order:
- Hhh, Assumed order=0, Hit: 20
- Fff, Assumed order=0, Hit: 12
- Jjj, Assumed order=0, Hit: 4
- Iii, Explicit order=1, Hit: 15
- Ggg, Explicit order=1, Hit: 10
- Eee, Explicit order=2, Hit: 11
- Ccc, Explicit order=3, Hit: 9
- Ddd, Explicit order=3, Hit: 8
- Bbb, Explicit order=4, Hit: 4
- Aaa, Explicit order=5, Hit: 14
Technical implementation approach
Previous technical remarks from @eric-basley:
SFS only contains needed data for search purpose, so SDMX annotation order is not included.
in order to re-conciliate both worlds (search & sdmx), we need to retrieve sdmx data from NSI servers.
Problems:
- SFS responses in 100ms, NSI responses in seconds
- Annotations are defined per dataflow whereas data-explorer request on many datasources !!
Solution is to enhance SFS to store orders related annotations consolidated for all indexed datasources.
Implementation:
- recompute orders each time a datasource or dataflow is added, updated or deleted.
- expose /api/annotations { [dimension names], locale } api that returns facet values orders.
REM: depending on required cpu time to compute orders from cached dataflow's structures we may update the cache asynchronously in batch modes.
Remark:
Maybe there would be a way to avoid storing the order information outside the search index, but instead include the order information into the internal facet item value similar to the current hierarchy information. The split of the information (hierarchy, item title, order) could then be done after retrieving facet values from the search before displaying the facet values.