Extend sfs log report method
As Frederic,
I want access to more specific and detailed log reports through filtering,
So that it's easier to understand which dataflows have been indexed, when they have been indexed, and by who they have been indexed (myself or by other users).
The goal is to functionally align the SFS log report method with the Transfer service log status method as spec'd here, so that both can be used by the DLM to display easily navigable log information from these services (see here).
As a dataspace admin or as any user, I want to be able to consult the status of all the indexation requests submitted and potentially filter the results by:
- Request ID (default: any)
- Tenant (organisationId) (default: any)
- User or group (by e-mail, default: any)
- Dataspace ID (default: any)
- Target Dataflow identification ("DATASPACE@AGENCY:ID(VERSION)", default: any) - allow filtering also for a dataflow indexed through the "dataflows" method
- Action
Search indexation
,Remove from search
orany
(default: any) - Submission time (datetime, default: 24h before now)
- Execution end time (datetime, default: now)
- Execution status (
Queued
,In progress
,Completed
, any; default: any) - whatever applies in this context - Outcome (
Success
,Warning
,Error
, any; default: any) - whatever applies in this context
The response should contain the following information per data upload request:
- Request ID
- Tenant (organisationId)
- All DataSources used
- User (email) performing the indexation (see #96 (closed))
- Dataspace ID - for indexations or deletes of specific dataflow otherwise "" for indexations or deletes of all dataflows
- Target Dataflow identification ("DATASPACE@AGENCY:ID(VERSION)") - for indexations or deletes of specific dataflow otherwise
All
for indexations or deletes of all dataflows - Action:
Search indexation
orRemove from search
- Submission time
- Execution start time (if different from submission time)
- Execution end time
- Execution status: Queued, In progress, Completed, TimedOut&Aborted - whatever applies in this context
- Outcome:
Success
,Warning
,Error
,None
,Multi-status
(when indexing all dataflows and some are successful and some failed) - When indexing more than one dataflow then provide the complete list of all to be indexed dataflows indicating:
- Execution start time (for indexing that dataflow)
- Dataspace ID
- Target Dataflow identification ("DATASPACE@AGENCY:ID(VERSION)")
- Status:
Success
,Warning
,Error
- Message: "Successfully indexed." or error message for user to understand why the action failed, e.g. provide the NSI error message
- Server: the name of the server (for developers/maintainers)
- Logger: the name of the component where the error occurred (for developers/maintainers)
- Exception: the full error exception message (for developers/maintainers), would e.g. also include the NSI HTTP error code
Technical Context
SFS produces reports for [POST /admin/dataflow, DEL /admin/dataflow, POST /admin/dataflows, DEL /admin/dataflows] requests
We would like to allow DLM users to access those reports data as log information.
All authenticated DLM users should have access to SFS log information.
Logs must be stored in persistent storage.
/admin
is used for POST and DELETE of dataflow(s), it continues to use api-key and will later be called by a trusted service bus client. We add a new api parameter for specifying the calling user’s email. With this, you would be able to log the user’s email.
/log
(new) is used for querying logs and will use a Keycloak token.
Todo
-
remove existing event driven report system and convert it to a log one where each event becomes a persistent log entry -
store all log entries in a mongodb database -
add a new api to request logs entries protected by a keycloak token. SFS will not check authorisation but just token validity. -
logs entries schema will follow #97 (closed), -
switch configs manager store from Redis to Mongodb (to avoid to use 2 persistent storages for SFS) -
add tests