Duplicate pipelines in Meltano UI after Extractor fails
While testing #1165 (closed) (check the comment thread here), I got a very weird bug with the pipeline for the Gitlab Extractor being duplicated after the extractor failed:
What I did:
-
Initialised a new project
-
Added a run the Gitlab Extractor with a wrong Group name --> It failed
-
Run the Extractor again with a wrong Project name --> It failed again
Somewhere between step 2 and 3 or immediately after step 3, the new pipeline above was created in the UI:
- It has the Gitlab Name
- It shows as succeeded
- It has no time on when it finished
- It has no logs of course (loading forever when I click on the log option)
Of course no pipeline has succeeded and if you click on the Reports option on either one of the pipelines, the models are still grayed out.
This seems like a pure UI bug as there is no additional pipeline reported in our logs:
sqlite> select * from job;
1|pipeline-1581077096062|FAIL|2020-02-07 12:04:58.583928|2020-02-07 12:05:02.078277|{"error": ""}|0|d4cd238d0e434ceca12124cc9d223710
2|pipeline-1581077096062|FAIL|2020-02-07 12:10:36.132317|2020-02-07 12:10:43.505225|{"error": ""}|0|e94b4edb4757461c8d62e9fe83d2074e
But it is not, as there is an entry for the new pipeline in meltano.yml
... ... ...
project_id: 0a235f0f-e068-46a7-8e2d-f26c74f30245
schedules:
- extractor: tap-gitlab
interval: '@once'
loader: target-postgres
name: pipeline-1581077096062
start_date: 2020-01-31 22:00:00
transform: run
- extractor: tap-gitlab
interval: '@once'
loader: target-postgres
name: pipeline-1581077433858
start_date: 2020-01-31 22:00:00
transform: run
And when I restart . Meltano UI, the additional pipeline is still there. I can start it manually and then it behaves as a normal pipeline, so I now have 2 pipelines :-)
Or should I make them 3? Yep, If I try to fix the connection and re-run the Tap, the first (bottom) pipeline runs again and a new one is added on top. And I can go like that and keep on adding more pipelines:
schedules:
- extractor: tap-gitlab
interval: '@once'
loader: target-postgres
name: pipeline-1581077096062
start_date: 2020-01-31 22:00:00
transform: run
- extractor: tap-gitlab
interval: '@once'
loader: target-postgres
name: pipeline-1581077433858
start_date: 2020-01-31 22:00:00
transform: run
- extractor: tap-gitlab
interval: '@once'
loader: target-postgres
name: pipeline-1581078835942
start_date: 2020-01-31 22:00:00
transform: run
- extractor: tap-gitlab
interval: '@once'
loader: target-postgres
name: pipeline-1581078932599
start_date: 2020-01-31 22:00:00
transform: run
- extractor: tap-gitlab
interval: '@once'
loader: target-postgres
name: pipeline-1581079022182
start_date: 2020-01-31 22:00:00
transform: run
So, to recap: Every time I try to fix the connection and restart, one more pipeline is added on top of everything else.