Transparency of the data retrieving process

Goal

Provide the user with transparency of the data retrieving process

TODO / How to implement

Display state and activity of the retrieval process in UI

DLE

  • detach the retrieval state from the DLE /status request and create a new API endpoint GET /instance/retrieval
  • extend the retrieval state and report its current value in API response GET /instance/retrieval 
{
  "mode": "logical", 
  "alerts": {}, 
  "lastRefresh": "2022-06-15T12:55:20Z", 
  "nextRefresh": null,
  "status": "refreshing", 
  "currentJob": "logicalRestore",
  "activity": {
    "source": [
      {
        "user": "postgres"
        "query": "<query text>" 
        "duration": "00:00:05.002151", 
        "wait_event_type": "Client",
        "wait_event": "ClientRead"
      }
    ],
    "target": [
      {
        "user": "postgres"
        "query": "<query text>" 
        "duration": "00:00:03.002843", 
        "wait_event_type": "Lock",
        "wait_event": "relation"
      }
    ]
  }
}

How to get activity

  • extend JobRunner interface with ReportActivity(context.Context) Activity
  • add a new field CurrentJob components.JobRunner to Retrieval.State
  • keep Retrieval.State.CurrentJob up-to-date 
    r.State.CurrentJob = job
...
r.State.CurrentJob = nil
  • implement for each job the ReportActivity method
    • both snapshot jobs (logicalSnapshot and physicalSnapshot) and physicalRestore do nothing
    • for logicalDump and logicalRestore set up PG connections to source and target respectively and get activity data
      • logicalDump uses config values
      • logicalRestore connects to Postgres in dblab_lr_-container
    • logicalDump in the immediateRestore mode fills both: source and target fields

Notes:

  • show activities only for logical mode and refreshing state
  • show activities both on source (if any – dump might be on S3) and target
    • skip if it's impossible to get activity
  • set a custom application_name (using the PGAPPNAME environment variable for pg_dump/pg_restore) - dle
  • prepare the query to show activity: 
      select
    usename as user,
    clock_timestamp() — query_start as duration,
    query,
    wait_event_type,
    wait_event
  from pg_stat_activity
  where application_name = 'dle'

UI

  • add a new client request GET /instance/retrieval to get retrieval state
  • add a new button next to status ("refreshing") – "Show details"
    • show activities only for logical mode and refreshing state
  • by clicking on the Show details button, display a modal pop-up, rendering two tables: "Activity on the source", "Activity on the target"
    • render tables side by side in two columns: left column - "Activity on the source", right - on the target
    • tables can have many rows
    • one or both tables may be empty (display proper message in this case)
    • render fields: query, duration, wait_event_type:wait_event in the tables
      • use the activity field from the API response
      • we might want to add more columns.
    • add a new button with icon to reload the state from the pop-up

Design with progress bars are not actual. Pay no attention to it

Acceptance criteria

The user sees the current state and activity of the retrieval process in the UI.

Edited by Lasha Kakabadze