Pipeline Name conflicts block creation of new pipelines
Setup
I am using the latest version from master (Jan 29) and testing locally.
I have initialised a new Meltano project to test all the recent changes.
Bug behaviour on initial run
I started meltano ui
and run a pipeline for the Gitlab Extractor.
Everything went fine, so I went and tried to add the Google Analytics Extractor, but got a pipeline already define error:
I was not sure about what was happening and whether that was an issue with the Google Analytics Extractor, so I went and tried to start a pipeline for the Stripe Extractor. I got the same error:
Note pipeline names in both screenshots: they are the same! Meltano UI is trying to use the same pipeline name here, which, of course conflicts with an existing pipeline. Which one? The one for the Gitlab extractor of course:
sqlite> select * from job;
1|pipeline-1580298791103|SUCCESS|2020-01-29 11:53:52.587094|2020-01-29 12:02:46.890373| ... ...
So for whatever reason, we are stuck in trying to use the first Pipeline name generated.
Bug behaviour on second run
So I decided to stop Meltano UI and start it again to check what happens.
Success! I was able to start a new pipeline for the Google Analytics Extractor! It got a new name and everything went according to plan.
The job log also shows that:
sqlite> select * from job;
1|pipeline-1580298791103|SUCCESS|2020-01-29 11:53:52.587094|2020-01-29 12:02:46.890373| ... ...
2|pipeline-1580299796056|SUCCESS|2020-01-29 12:10:11.120259|2020-01-29 12:11:00.986985|{}|0|996b4d6b329145a78b7d397e6f439e3c
But then I tried to start a pipeline for the Stripe Extractor and things got really weird:
- it seems that Meltano UI is still trying to use the same pipeline name as the first one in this run
- I did not get an error this time around but instead the pipeline for Google Analytics Switched to showing that it was for Stripe!
- Checking the logs for the pipeline show that it was indeed run for Google Analytics. So it is more of a UI thing
- The
Connect
button for Stripe was replaced by theView Pipeline
one as the pipeline for Google Analytics is now registered as its own pipeline. That means that I can not start a pipeline for Stripe - The reverse is true for Google Analytics. It now shows a
Connect
option next to it even though a pipeline has run
And if I click Connect
to Google Analytics it switches the pipeline back to it.
And then if I click Connect
on Stripe, it switches the pipeline again and on and on and on ...
Third Time's the charm
So I decided to stop Meltano UI and start it again to check what happens.
I added successfully and run a pipeline for the Facebook Ads Extractor.
And then I got the same behaviour when trying to start Stripe, only this time around, it switches places with the last pipeline run (the one for Facebook Ads)
So it is clear that each time we restart Meltano UI we get a good new pipeline name that we can use once to start a pipeline for an Extractor and then we get stuck with it and can not start another pipeline.
Success
Restart and all over again to add and run Stripe --> Success
Note that whenever we restart Meltano UI, the previous pipelines are properly set to their proper extractor even if they were messed up before shutting down Meltano UI. This also indicates that this is a bug on the front end layer