Multiple jobs run at the same time for hourly scheduled pipeline
Edit: Updated the title to reflect the real issue identified after my investigation. Leaving the original description for context
This is an investigation on whether we have a bug with our pipeline setup flow.
Description of behavior witnessed
When: During initial setup of first pipeline
Related issue: meltano/account-management#1
First time on instance (after initial setup)
- User installed Stripe Extractor and setup the configuration.
- User clicked on
Test Connection--> Extractor was reported as working
- User skipped Loader setup (was set during instance setup)
- User selected to run Transforms for Stripe
- A pipeline started --> it was taking too much time to wait during the call, so we left it running
- @dmor went back after the call (while the pipeline was still running)
- initially, @dmor got an error while trying to access the Analyze page for one of Stripe's designs
- Afterwards, data returned even though the pipeline was still running (and the transforms have not run)
Strange behavior --> The default report had data in there
Expected behavior --> There should be no data in there before the transforms run
Guess of what happened
The only reasonable explanation I can think of, is that during the
Test Connection, the job that started was never killed and it brought data in, run the transformations and populated the
analytics tables that are used for the Meltano Analysis reports.
We have to investigate if this is the case and address this issue --> it could affect the behavior of extractors and pipelines when users click the
Test Connection button