Problem installing a custom transform and model when upgrading version 1.19.0 to 1.2X
Hi! I tried different things to make it works with version 1.19.0 and it did not work when I upgrade the version to 1.22.1. First of all, I created github repository for the transform, the model and the dbt. I install custom extractor by doing this command:
meltano add --custom extractor tap-github
(namespace): tap_github
(pip_url): tap-github==1.5.0 or git+https://github.com/L3houx/tap-github.git
(executable) [tap-postgres]: tap-github
(capabilities): discover,properties,state
This is my meltano.yml file. I added settings to the extractor to use env variables. I also added a custom transform and model.
plugins:
dashboards:
- name: dashboard-google-analytics
pip_url: git+https://gitlab.com/meltano/dashboard-google-analytics.git
extractors:
- capabilities:
- discover
- properties
- state
executable: tap-github
name: tap-github
namespace: tap_github
pip_url: git+https://github.com/L3houx/tap-github.git
settings:
- env: GITHUB_A_T
name: access_token
- env: GITHUB_REPO
name: repository
- label: GitLab
name: tap-gitlab
pip_url: git+https://gitlab.com/meltano/tap-gitlab.git
- label: Google Analytics
name: tap-google-analytics
pip_url: git+https://gitlab.com/meltano/tap-google-analytics.git
loaders:
- label: PostgreSQL
name: target-postgres
pip_url: git+https://github.com/meltano/target-postgres.git
models:
- name: model-github
pip_url: git+https://github.com/L3houx/model-github.git
- name: model-gitlab
pip_url: git+https://gitlab.com/meltano/model-gitlab.git
- name: model-google-analytics
pip_url: git+https://gitlab.com/meltano/model-google-analytics.git
orchestrators:
- name: airflow
pip_url: apache-airflow==1.10.2
transformers:
- name: dbt
pip_url: dbt==0.14.4
transforms:
- name: tap-github
pip_url: https://github.com/L3houx/test-dbt-tap-github.git
vars:
schema: '{{ env_var(''MELTANO_LOAD_SCHEMA'') }}'
- name: tap-gitlab
pip_url: https://gitlab.com/meltano/dbt-tap-gitlab.git
vars:
schema: '{{ env_var(''MELTANO_LOAD_SCHEMA'') }}'
ultimate_license: '{{ env_var(''GITLAB_API_ULTIMATE_LICENSE'', False) }}'
- name: tap-google-analytics
pip_url: https://gitlab.com/meltano/dbt-tap-google-analytics.git
vars:
schema: '{{ env_var(''MELTANO_LOAD_SCHEMA'') }}'
project_id: 3bd1afa6-b687-4b12-8b8d-595c547b1215
send_anonymous_usage_stats: true
version: 1
After that, I tried to do :
- meltano add transform tap-github
- meltano add model model-github
- meltano install and nothing works.
So I tried to had the section into discovery.yml. So I added this to discovery.yml in the transforms section:
- name: tap-github
namespace: tap_github
pip_url: 'https://github.com/L3houx/test-dbt-tap-github.git'
vars:
schema: "{{ env_var('MELTANO_LOAD_SCHEMA') }}"
I added this to discovery.yml in the models section:
- name: model-github
namespace: tap_github
pip_url: git+https://github.com/L3houx/model-github.git
After I did the same 3 commandes:
- meltano add transform tap-github
- meltano add model model-github
- meltano install
The custom transform and model installed and it worked when I get data from a github repsitory. The problem came when I upgrade meltano's version with the command: meltano upgrade
. Its uninstalled the transform and the model and every time I tried to add sections to discovery, it is reloading to the original version and erase all the modification I made.
I also tried: meltano add --custom transform tap-github
and this is the error I got:
I would like to know how to properly add a custom transform and model.
Thank you and have a nice day!