[Observes] automatic dynamo ETL
Problem to solve
Dynamo ETL relies on a manual step for finalizing. i.e.
- The creation of the loading schema
- The centralization/migration of the loading schema into the core schema
The main issue for achieving this is that the ETL is distributed and therefore the event of finalization of the loading process does not trigger anything.
Intended users
Permissions and Security
Proposal
The ideal is to trigger the required jobs as soon as the loading process finish, but this requires support from makes
. For simplifying the model, a job can be scheduled at the expected hour of finalization.
The job will:
- check success or failure of executed parts of the ETL
- perform retries of failed job parts due to possible unhandled errors (see #7846).
- perform the centralization procedure.
What does success look like, and how can we measure that?
- Automatic execution of the full dynamo ETL process.
Links / references
/label typefeature