1. Custom pipelines built in [Python](/handbook/enterprise-data/platform/python-guide/) and orchestrated via [Airflow](/handbook/enterprise-data/platform/infrastructure/)
1. Flows built in Tableau Prep and orchestracted by Tableau Cloud
@@ -56,14 +56,14 @@ It is essential to assess current and future requirements for the data because t
ETL Vendors, like Fivetran give us more flexibility than a Snowflake share, but they come with additional cost to our contract. As noted, these can be a great option when we need to move quickly, especially when the criticality of a new data source is unclear.
A lack of flexibility and maintainability is still an important consideration here. We've had trouble managing high complexity pipelines within vendor interfaces, and have experienced pain in change management without the ability to apply approvals or tests to changes. If the complexity of objects and/or attributes is relatively high, it might be worth considering a custom pipeline. Another aspect to consider here is that ETL vendors are often [limited in the scheduling of the refreshes](https://gitlab.com/gitlab-data/analytics/-/issues/21649#note_2262432625)
A lack of flexibility and maintainability is still an important consideration here. We've had trouble managing high complexity pipelines within vendor interfaces, and have experienced pain in change management without the ability to apply approvals or tests to changes. If the complexity of objects and/or attributes is relatively high, it might be worth considering a custom pipeline. Another aspect to consider here is that an ETL vendor is often [limited in the scheduling of the refreshes](https://gitlab.com/gitlab-data/analytics/-/issues/21649#note_2262432625)
The volume of records is also a primary consideration for using Vendors at the moment as they are usually priced on usage. Contract impact is a required evaluation step for this solution. High cost may also warrant a custom pipeline.
Some data sources are just too sensitive to allow for a third party to have access. Keeping the pipeline within our data platform is likely required for such cases.
For Platform efficiency, we do not want vendors in this area with overlapping functionalities. This means we want a maximum of 1 vendor. While vendors often provide different sets of available connectors, the availability of connectors in a different vendor's product is not sufficient justification to add that vendor. A business case to add a new vendor to the Data Platform must include replacing the current existing vendor.
Currently, the Data Platform has multiple vendors (Fivetran, Stitch, and Meltano). This means we will consolidate to 1 vendor (to be determined).
Currently, the Data Platform has 1 vendor: Fivetran.
@@ -7,7 +7,7 @@ On this page the main intention is to keep all details about the inventory we ar
The motivation is to keep a neat inventory list with details and criteria of who, why, when and what we will upgrade. Also, the plan is to expose detailed conditions regarding the upgrade and command line application to gather all details about the recent and latest versions of Python libraries. The inventory list contains:
- 🐍 `Python` versions in use
- 🛠 Tools we are using (`dbt`, `airflow`, `permifrost`, `meltano`)
- 🛠 Tools we are using (`dbt`, `airflow`, `permifrost`)
- 📚 Python libraries _(packages)_ in use (i.e `pandas`, `requests`, `matplotlib`)
We strive to have a DRI for every tool or package. The DRI responsibility is to keep that tool or package in a healthy shape by advice and or initiate upon upgrades.
@@ -63,7 +63,6 @@ There are several images we are using 🐍`Python`. Various versions are in use
@@ -160,7 +157,6 @@ This is because anything becomes deprecated usually no longer receives any bug f
| [dbt](/handbook/enterprise-data/platform/dbt-guide/) | `1.9.2` | [link](https://docs.getdbt.com/docs/dbt-versions/core#latest-releases) | - [dbt best practices for upgrading](https://docs.getdbt.com/docs/dbt-versions/core#best-practices-for-upgrading)<br>- [Upgrading dbt version](https://gitlab.com/gitlab-data/runbooks/-/blob/main/infrastructure/upgrading_dbt_version.md) | `TBA` | - `Data Platform`<br>- `Analytics Engineers`<br>- `Data Scientists` | Not more than 2 versions behind (beta release excluded) and minimum support level `critical` |
| [airflow](/handbook/enterprise-data/platform/infrastructure/#airflow) | `2.10.5` | [link](https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html#version-life-cycle) | [Upgrade Plan for Airflow](https://gitlab.com/gitlab-data/analytics/-/work_items/26008) | `TBA` | `Data Platform` | Current version released > 1 year |
| [permifrost](/handbook/enterprise-data/platform/permifrost/) | `0.15.4` | [link](https://gitlab.com/gitlab-data/permifrost) | [Upgrading permifrost version](https://gitlab.com/gitlab-data/permifrost/-/blob/master/RELEASE.md?ref_type=heads) | @rbacovic | `Data Platform` | Not more than 2 versions behind (beta release excluded) |
| [meltano](/handbook/enterprise-data/platform/Meltano-Gitlab/) | `2.16.1` | [link](https://github.com/meltano/meltano/releases) | [Upgrade Meltano version](https://gitlab.com/gitlab-data/gitlab-data-meltano/-/merge_requests/34) | `TBA` | `Data Platform` | Current version released > 1 year |
