NSI uses wrong period end of the filter's endPeriod parameter & high-frequency values are ignored- SDMXRI-1730 - NSIWS v8.5.0
Wrong EndPeriod treatment
NSI data retriever doesn't expand passed endPeriod till the end of the represented timeframe. The impact is that the NSI doesn't return all time periods that correspond to the time range indicated in the time period filter. For example when rest query contains endPeriod=2020 search filter in SQL query uses value of 2020-01-01 while it should use 2020-12-31 23:59:59.
It means daily data fails to be returned when filtering on a single year. Filtering on a single year (including a daily data) returns the message "There is no data available".
Steps to reproduce
- This example in QA was uploaded on a dataflow with Frequency=Annual and Date format=daily data: https://de-qa.siscc.org/vis?lc=en&df[ds]=qa%3Areset&df[id]=DF_E5301_TEST&df[ag]=LU1&df[vs]=1.0&av=true&pd=2015%2C2020&dq=.A
- When filtering on 2010-2015, it correctly returns daily Time-Period data, but filtering on a single year e.g. 2020-2020, it returns no data even though it should return data for 2020-01-01 : https://de-qa.siscc.org/vis?lc=en&df[ds]=qa%3Areset&df[id]=DF_E5301_TEST&df[ag]=LU1&df[vs]=1.0&av=true&pd=2020%2C2020&dq=.A
From a first analysis on the JavaScript side, it seems that the problem is on the .NET side. The following request is properly set, but it returns a 404
: https://nsi-qa-reset.siscc.org/rest/data/LU1,DF_E5301_TEST,1.0/.A?startPeriod=2020&endPeriod=2020&dimensionAtObservation=AllDimensions
High-frequency time period values are ignored when filtering for startPeriod and endPeriod
High-frequencies are those above the day (hourly, minutely or secondly).
Tested with the dataflow "Key Short-Term Economic Indicators" in qa:reset
(in this dataspace for now because there are no conflict with or references to other DSDs/DFs) using these files:
OECD-DF_KEI-1.0-all.xml
OECD-DF_KEI-1.0-data-slice.zip
The DSD has the proper new annotation "type": "SUPPORT_DATETIME"
, and there are data for different hourly and minutely frequencies, with various time period formats (YYYY-MM-DDThh:mm:ss and YYYY-MM-DDThh:mm:ss/PnY).
- For Hourly data, when filtering on frequency=Hourly, the start hour period isn't taken into consideration, e.g. when I filter for startPeriod : Year=2006 Month=01 Day=15 Hour=03:00, I still retrieve in the table values for Time=2006-01-15T01:00:00. https://de-qa.siscc.org/vis?lc=en&df[ds]=qa%3Areset&df[id]=DF_KEI&df[ag]=OECD&df[vs]=1.0&av=true&pd=2006-01-15T03%3A00%3A00%2C&dq=...H&ly[cl]=TIME_PERIOD
- For Minutely data, when filtering on frequency=Minutely, the start minute period isn't taken into consideration, e.g. when I filter for startPeriod : Year=2007 Month=04 Day=12 Hour=03:00 Minute=03, I still retrieve in the table values for Time=2007-04-12T03:02:00. https://de-qa.siscc.org/vis?lc=en&df[ds]=qa%3Areset&df[id]=DF_KEI&df[ag]=OECD&df[vs]=1.0&av=true&pd=2007-04-12T03%3A03%3A00%2C&dq=...N&ly[cl]=TIME_PERIOD
- Eurostat ticket: https://citnet.tech.ec.europa.eu/CITnet/jira/browse/SDMXRI-1730
- Eurostat's pull request: https://citnet.tech.ec.europa.eu/CITnet/stash/projects/SDMXRI/repos/maapi.net/pull-requests/75/overview