Persist and consider state messages regardless of job success
Problem to solve
So a usecase I have is the Github tap which will take days to run for the first time (cold start). Every time, this process breaks for some reason, it is super expensive and frustrating to restart it from scratch even though the work is done; i.e. I have the data in the target.
Target audience
Meltano ELT users
Further details
- Cold Start
- Incremental Runs
Proposal
I think storing all the state messages outputted by a tap during a run will be valuable. https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md#state-message these are structured logs and can be ingested in meltano db or some other store for future use.
What does success look like, and how can we measure that?
DB stores all the states logs in a table which can be queried to start new runs more efficiently.
Edited by Douwe Maan