Improve and harmonise identification of frequency and ref_area dimension
System dimensions are dimensions that have an impact on the behavior of the app:
- time axis in timeline chart
- frequency for filters
- ref area for choropleth map
These dimensions require to be identified properly:
- role
- concept id
- dimension id
Today, the identification is not completely homogeneous everywhere and needs to be refactored and applied identically everywhere (ie double checks specs, update sdmxjs, use sdmxjs).
Rules that are used to identify these dimensions rely on mechanism that could be generic (check id, then role etc...) and user data that should be configurable (valid ref area ids are COU, COUNTRY etc...).
While working on refactoring identification, it could be nice to introduce this flexibility.
The issue is often caused by the fact that the current parsing mechanism cannot identify clearly both the Time Period dimension and the current Frequency, e.g. when the FREQUENCY dimension as well as the TIME_FORMAT attribute are absent or empty.
The Frequency identification mechanism should be enhanced and made as much as possible coherent with the Time Period filter parsing mechanism for the visualisation page (see: #168 (closed)), while a main difference remains in the fact that the filter uses structure messages while the chart uses data messages.
The Frequency identification mechanism should be:
1. Retrieval of the current frequency (https://oecd.aha.io/features/ONEDATA-32):
-
The frequency dimension can be recognised by its role: "FREQ" (see: https://sdmx.org/?sdmx_news=just-published-guidelines-for-sdmx-concept-roles). If such role is not available/found then by its concept ID or by its dimension ID: "FREQ" or "FREQUENCY".
-
If found then the chart uses the currently (first) selected frequency in the data selection. Most used frequencies are:
frequency code ID | meaning |
---|---|
A2 | Biennial |
A | Annual |
S | Half-yearly, Semester |
Q | Quarterly |
M | Monthly |
W | Weekly |
D | Daily |
H | Hourly |
B | Daily - business week |
N | Minutely |
2. If no frequency dimension is found, then retrieval of the time-format attribute.
-
This attribute will implicitly indicate the frequency assigned for the data of the dataset. The time format attribute can be recognised by its concept ID or by its dimension ID: "TIME_FORMAT".
-
If found then the chart uses the currently (first) retrieved time format value. Most used time formats are:
time format code ID | meaning |
---|---|
P1D | Daily (or Daily - business week) |
P1M | Monthly |
P1Y | Annual |
P3M | Quarterly |
P6M | Half-yearly, Semester |
P7D | Weekly |
PT1M | Minutely |
3. If no frequency dimension nor time-format attribute are found, then retrieval of (the first 2 time periods in) the time dimension (only for data messages that are not partial).
-
The time period dimension can be recognised by its role: "TIME_PERIOD". If such role is not available/found then by its concept ID or by its dimension ID: "TIME_PERIOD" or for backward-compatibility also "TIME", "YEAR".
-
If found then the first two retrieved time period values are used and the duration between their (start) dates is to be calculated, e.g.
0: {start: "2009-01-01T00:00:00Z", end: "2009-12-31T23:59:59Z", id: "2009"}
1: {start: "2010-01-01T00:00:00Z", end: "2010-12-31T23:59:59Z", id: "2010"}
duration = "2010-01-01T00:00:00Z" - "2009-01-01T00:00:00Z" = 356d ==> Annual or
duration = "2010" - "2009" = 356d ==> Annual
duration | meaning |
---|---|
1d | Daily (or Daily - business week) |
28d - 31d | Monthly |
356d - 366d | Annual |
89d - 92d | Quarterly |
178d - 188d | Half-yearly, Semester |
7d | Weekly |
1mn | Minutely |
4. If no frequency dimension, time-format attribute and time period dimension are found, then the time period/frequency filter and the timeline chart menu option are to be disabled/hidden.
Example usecase: http://de-staging-siscc.redpelicans.com/vis?dataquery=AUS.SOX%2BNOX.TOT&period=2010%2C2018&frequency=A&locale=en&facet=oecdcs1&constraints[0]=oecdcs1%2CEnvironment%23ENV%23&start=0&dataflow[datasourceId]=staging:SIS-CC-stable&dataflow[dataflowId]=AIR_EMISSIONS_DF&dataflow[agencyId]=OECD&dataflow[version]=1.0&viewer=TimelineChart&filter=PANEL_PERIOD
where the Timeline doesn't render anything.
The Geo/Ref_Area dimension identification mechanism should be:
-
The Ref_Area dimension can be recognised by its role: "GEO" (see: https://sdmx.org/?sdmx_news=just-published-guidelines-for-sdmx-concept-roles) or for backward-compatibility also "REF_AREA". Note that several dimensions could have the role "GEO" thus only the first of those dimensions is to be used.
-
If such a role is not available/found then identify the dimension by its concept ID or by its dimension ID: "REF_AREA" or for backward-compatibility also "COUNTRY", "LOCATION", "REGION", "REFERENCE_AREA".
If no Ref_Area dimension is found, then the map chart menu option is to be disabled/hidden.
--> Use case: Identify the Geo/Ref_Area dimension for the Choropleth map: The configured Choropleth maps should propose as options in the DE Viz Chart menu only, if the currently displayed dataflow contains a Ref_Area dimension identified through the above rules.