Update data team handbook for data-tests consolidation

---

This page documents the CI jobs used by the data team in Merge Requests in both the [Data Tests](https://gitlab.com/gitlab-data/data-tests) and [Analytics](https://gitlab.com/gitlab-data/analytics) projects. For a quickstart guide on how to get started doing Merge Requests and using CI jobs, see [this practical guide](/handbook/enterprise-data/how-we-work/practical-guide/) in How We Work.

This page documents the CI jobs used by the data team in Merge Requests in the [Analytics](https://gitlab.com/gitlab-data/analytics) project. For a quickstart guide on how to get started doing Merge Requests and using CI jobs, see [this practical guide](/handbook/enterprise-data/how-we-work/practical-guide/) in How We Work.

## What to do if a pipeline fails

#### `clone_stop`

Runs automatically when MR is merged or closed. In general, this should not be run manually but it can also be used to remove the MR development database. Once completed, starting a new pipeline or re-running the `clone_prod` job will then re-clone the empty database and you can start again

## Data Test Pipelines

All the below run against the Prod DB using the changes provided in the repo. No cloning is needed to run the below.

### `🧠 all_tests_prod`

Runs through all tests in the analytics & data tests repo.

### `💾 data_tests_prod`

Runs through all the data tests in the analytics & data tests repo's.

### `schema_tests_prod`

Runs through all the schema tests in the analytics & data tests repo's.

### `specify_tests_prod`

Runs specified model tests with the variable `DBT_MODELS`

- Ensure that you have updated your `.dbt/profiles.yml` with your specific user configuration

- Ensure your SSH configuration is setup according to the [GitLab directions](https://docs.gitlab.com/user/ssh/). Your keys should be in `~/.ssh/` and the keys should have been generated with no password.

  - You will also need access to [this project](https://gitlab.com/gitlab-data/data-tests) to run `dbt deps` for our main project.

- **NB**: Ensure your default browser is set to chrome. The built-in SSO login only works with chrome

- **NB**: Ensure you are in the folder where your `/analytics` repo is located. If you installed everything properly `jump analytics` will land you where it is needed in order to run `dbt` commands successfully.

- **NB**: Before running dbt for the first time run `make prepare-dbt`. This will ensure you have venv installed.

    ```

- To start a `dbt` container and run commands from a shell inside it, use `make run-dbt`. This command will install or update the dependencies required for running dbt.

- To start a `dbt` container without the dependency update use `make run-dbt-no-deps`. This command assumes you already have the dbt dependencies installed. When using this command, if you make changes in any of the dependency packages (e.g. data-tests), you will need to run either `dbt deps` (from within the shell) or `make run-dbt` again for these changes to show up in your repository.

- To start a `dbt` container without the dependency update use `make run-dbt-no-deps`. This command assumes you already have the dbt dependencies installed. When using this command, if you make changes in any of the dependency packages, you will need to run either `dbt deps` (from within the shell) or `make run-dbt` again for these changes to show up in your repository.

- This will automatically import everything `dbt` needs to run, including your local `profiles.yml` and repo files

- To see the docs for your current branch, run `make run-dbt-docs` and then visit `localhost:8081` in a web-browser. Note that this requires the `docs` profile to be configured in your `profiles.yml`

  "terminal.integrated.env.osx": {

  "SHELL":"/bin/zsh",

  "DBT_PROFILES_DIR": "../../../.dbt/",

  "DATA_TEST_BRANCH":"main",

  "SNOWFLAKE_PROD_DATABASE":"PROD",

  "SNOWFLAKE_PREP_DATABASE":"PREP",

  "SNOWFLAKE_SNAPSHOT_DATABASE":"SNOWFLAKE",

Tags can be added in YAML files or in the config settings of any model. Review the [`dbt_project.yml`](https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/dbt_project.yml) file for several examples of how tags are used. Specific examples of adding tags for the [Trusted Data Framework](/handbook/enterprise-data/platform/dbt-guide/#tagging) are shown below.

Within the `analytics` and `data-tests` projects we enforce a Single Source of Truth for all tags. We use the [Valid Tags CSV](https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/data/valid_tags.csv) to document which tags are used. Within merge requets, there is a validation step within every dbt CI job that will check this csv against all tags used in the project and fail the job if there is a mismatch. In the future, we aim to include more metadata about tags within this csv file.

Within the `analytics` project we enforce a Single Source of Truth for all tags. We use the [Valid Tags CSV](https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/data/valid_tags.csv) to document which tags are used. Within merge requets, there is a validation step within every dbt CI job that will check this csv against all tags used in the project and fail the job if there is a mismatch. In the future, we aim to include more metadata about tags within this csv file.

Be aware that tags applied at any level do not apply to any tests. Tags for tests have to be explicitly applied for every test within the `schema.yml` file.

1. [Rowcount tests](/handbook/enterprise-data/platform/dbt-guide/#rowcount-tests) to determine if the number of rows in a table over a pre-defined period of time match pre-defined thresholds or literals

1. [Custom SQL tests](/handbook/enterprise-data/platform/dbt-guide/#custom-sql) any valid SQL that doesn't conform to the above categories

Our tests are stored in 2 primary places - either in a YAML file within our [main project](https://gitlab.com/gitlab-data/analytics) or within our [Data Tests](https://gitlab.com/gitlab-data/data-tests) project.

Our tests are all stored in the [analytics](https://gitlab.com/gitlab-data/analytics) project.

Schema and Column Value tests will usually be in the main project. These will be in `schema.yml` and `sources.yml` files in the same directory as the models they represent.

Schema and Column Value tests will usually be in `schema.yml` and `sources.yml` files in the same directory as the models they represent.

Rowcount, and any other custom SQL tests will always be in the [Data Tests](https://gitlab.com/gitlab-data/data-tests) project. This is a private project for internal GitLab use only.

Rowcount and any other custom SQL tests live in [`transform/snowflake-dbt/tests/`](https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt/tests), organized by business domain.

##### Tagging

This test is implemented as a [dbt macro](https://docs.getdbt.com/docs/build/jinja-macros). This means that instead of writing SQL, a user can add the test by simply calling the macro. This is controlled by the [`source_rowcount`](https://dbt.gitlabdata.com/#!/macro/macro.gitlab_snowflake.source_rowcount) macro.

```sql

-- https://gitlab.com/gitlab-data/data-tests/-/blob/main/tests/sources/zuora/rowcount/zuora_subscription_source_rowcount_2019.sql

-- https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/tests/sources/zuora/rowcount/zuora_subscription_source_rowcount_2019.sql

{{ config({

    "tags": ["tdf","zuora"]

    })

Purpose: We have a fast-growing business and should always have at least 50 and at most 200 new Subscriptions loaded from the previous day. This is controlled by the [`model_new_records_per_day`](https://dbt.gitlabdata.com/#!/macro/macro.gitlab_snowflake.model_new_rows_per_day) macro.

```sql

-- https://gitlab.com/gitlab-data/data-tests/-/blob/main/tests/sources/zuora/rowcount/zuora_subscription_source_model_new_records_per_day.sql

-- https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/tests/sources/zuora/rowcount/zuora_subscription_source_model_new_records_per_day.sql

{{ config({

    "tags": ["tdf","zuora"],

    "severity": "warn",

#### Merge Request Workflow

There are a few scenarios to think about when adding or updating tests.

All tests — whether YAML-based or custom SQL — live in the `analytics` repo, so there is only ever one MR to open.

The first scenario is modifying or adding a test in a YAML file within our main project. This follows our standard MR workflow and nothing is different. Run the [CI jobs](/handbook/enterprise-data/platform/ci-jobs) as you normally would.

The second scenario is adding any test or golden data records within the `data-tests` project against tables that are being updated or added via an MR in the `analytics` project. This is the most common scenario. In this case, no pipelines need to be executed in the `data-tests` project MR. The regular dbt pipelines in the `analytics` MR can be run and the only change is the branch name of the `data-tests` project needs to be passed to the job via the `DATA_TEST_BRANCH` environment variable.

The third scenario is when tests are being added to the `data-tests` project, but no golden data CSV files are being updated or added, AND there is no corresponding MR in the `analytics` project. In this scenario, you will see some CI jobs that will run your tests against production data. This is useful for ensuring minor changes (such as syntax, tags, etc.) work.

The fourth scenario is when golden data CSV files are being added or updated and there is no corresponding `analytics` MR. In this case, we do not want to test against production as the golden data files are inserted into the database as tables. In this scenario, you will see additional CI jobs. There are some to create a clone of the warehouse, some to run dbt models stored in the `analytics` project against the clone, and some to run the tests against the clone.

This flowchart should give a rough guide of what to do. Follow the instructions in the relevant MR template in the project for more detailed instructions.

```mermaid

graph TD

  A[Do you want to add YAML or SQL tests?] -->|YAML| B[Open an Analytics MR <br/>and proceed as usual]

  A -->|SQL| C[Open a Data Tests MR]

  A -->|Both| D[Open an Analytics and Data Tests MR]

  D --> E[Run dbt jobs as usual in analytics <br/>but provide the branch name of the <br/> Data Tests project as `DATA_TEST_BRANCH`]

  C --> F[Are you updating or adding<br/> a Golden Data CSV?]

  F -->|Yes| G[Run a clone - full or shallow - then run dbt jobs]

  F -->|No| H[Run dbt jobs against production]

```

In the case where you have a merge request in `data-tests` and one in `analytics`, the `analytics` [MR should be set as a dependency](https://docs.gitlab.com/ee/user/project/merge_requests/dependencies.html) of the `data-tests` MR. This means that the `analytics` MR must be merged prior the `data-tests` MR being merged.

#### Running the newly introduced dbt tests in the data-tests project

Steps to follow in order to run the tests you implemented in the data-tests project from your machine, while developing them:

1. Push your changes to the remote branch you are working on the data-tests project

2. Go to your `analytics` project locally, create a new branch (`git checkout -b <branch_name>`) with the same name as the one at `data-tests` & modify the `Makefile` to edit the `DATA_TEST_BRANCH` to match your branch name on the `data-test` project

3. From the `analytics` project run `make run-dbt`

4. You should see some logs, which also show the revision data-tests was installed from, where you should see your branch

5. From where you currently are (which should be the `snowflake-dbt` directory) run the corresponding command for testing your own model

- **Adding or modifying YAML tests** (`schema.yml` / `sources.yml`): follow the standard MR workflow and run the [CI jobs](/handbook/enterprise-data/platform/ci-jobs) as usual.

- **Adding or modifying custom SQL tests**: add the SQL file(s) to [`transform/snowflake-dbt/tests/`](https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt/tests) in the same `analytics` MR. Run the standard dbt CI jobs to validate.

#### Example

To run the `zuora_revenue_revenue_contract_line_source` rowcount tests, we can use the following command, which should work without any issues:

To run the `zuora_revenue_revenue_contract_line_source` rowcount tests, use the following command from within the venv shell:

`dbt --partial-parse test --models zuora_revenue_revenue_contract_line_source`

> :warning:  Please note, whenever you make changes to the underlying tests in the data-tests project, you need to push those changes to the remote and re-run steps 3-5,  to start a dbt container with the latest changes from your branch.

##### Data extraction (RAW data layer)

Data extraction is loading data from the source system to Snowflake data warehouse in the RAW data layer.

| package name                                                                                | Version in use |  DRI  | User                                   |

|---------------------------------------------------------------------------------------------|----------------|-------|----------------------------------------|

| [snowflake_spend](https://gitlab.com/gitlab-data/snowflake_spend)                           | `1.1`          | `N\A` |-Data Engineers<br>-Analytics Engineers |

| [data-tests](https://gitlab.com/gitlab-data/data-tests)                                     | `N\A`          | `N\A` |-Data Engineers<br>-Analytics Engineers |

| [dbt-labs/audit_helper](https://github.com/dbt-labs/dbt-audit-helper)                       | `0.9.0`        | `N\A` |-Data Engineers<br>-Analytics Engineers |

| [dbt-labs/dbt_utils](https://github.com/dbt-labs/dbt-utils)                                 | `1.1.1`        | `N\A` |-Data Engineers<br>-Analytics Engineers |

| [dbt-labs/snowplow](https://github.com/dbt-labs/snowplow/tree/0.15.1/)                      | `0.15.1`       | `N\A` |-Data Engineers<br>-Analytics Engineers |

For more details how the program if running, refer to the [**source code**](https://gitlab.com/gitlab-data/package_inventory/-/tree/master/README.md).

**Note:** Keep in mind that we are relying on [PyPi](pypi.org) and [GitLab Data](https://gitlab.com/groups/gitlab-data) group as a primary source of the latest version of the library.

**Note:** Keep in mind that we are relying on [PyPI](https://pypi.org) and [GitLab Data](https://gitlab.com/groups/gitlab-data) group as a primary source of the latest version of the library.

> **Table 3:** DRI for Python libraries

🛑 **Dont`s**:

- Do not upgrade to the `pre-release` version of the software, always use a [stable release](https://en.wikipedia.org/wiki/Software_release_life_cycle#Stable_release) version

- Do not use `non-trusted` sources. We prefer [PyPi](pypi.org) or packages from [GitLab Data](https://gitlab.com/groups/gitlab-data) group for the source of the installation

- Do not use `non-trusted` sources. We prefer [PyPI](https://pypi.org) or packages from [GitLab Data](https://gitlab.com/groups/gitlab-data) group for the source of the installation

### Upgrade logging