Remove stale content from dbt guide and update Makefile command reference

#### Venv Workflow {#Venv-workflow}

Recommended workflow for anyone running a Mac system.

The recommended workflow for Mac users. All dbt commands are run inside a Poetry-managed virtual environment so every developer is on the same locked dbt version and dependencies.

#### Using dbt

- Ensure you have the `DBT_PROFILE_PATH` environment variable set. This should be set if you've used the [onboarding_script.zsh](https://gitlab.com/gitlab-data/analytics/-/blob/master/admin/onboarding_script.zsh) (recommened to use this as this latest and updated regularly), but if not, you can set it in your `.bashrc` or `.zshrc` by adding `export DBT_PROFILE_PATH="/<your_root_dir/.dbt/"` to the file or simply running the same command in your local terminal session

- Add the following snippet to your `.bashrc` or `.zshrc` to set up pyenv. This defines where pyenv is installed (through the onboarding script), adds it to your path, and runs pyenv's initialization script to set up shell integration to calls to python and route them to the correct pyenv-managed version.

 ```bash

- **NB**: Ensure your default browser is set to chrome. The built-in SSO login only works with chrome

- **NB**: Ensure you are in the folder where your `/analytics` repo is located. If you installed everything properly `jump analytics` will land you where it is needed in order to run `dbt` commands successfully.

- **NB**: Before running dbt for the first time run `make prepare-dbt`. This will ensure you have venv installed.

  - This will run a [series of commands](https://gitlab.com/gitlab-data/analytics/-/blob/master/Makefile#L111-114) including downloading and running a `poetry` install script.

  - This will download and run a `poetry` install script to set up the virtual environment.

  - If you get a certificate error like this `urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1124)>`, follow these [StackOverflow instructions](https://stackoverflow.com/questions/50236117/scraping-ssl-certificate-verify-failed-error-for-http-en-wikipedia-org/53310545#53310545).

  - If you get an error like this `ssh: connect to host gitlab.com port 22: Operation timed out fatal: Could not read from remote repository.`, and you've made sure you have correct access rights and the repository exists, then it could be an SSH issue where SSH connections to GitLab on port 22 were being blocked by network/firewall. To solve this, configure SSH to use GitLab's alternative SSH service on port 443.

We use virtual environments for local dbt development because it ensures that each developer is running exactly the same dbt version with exactly the same dependencies. This minimizes the risk that developers will have different development experiences due to different software versions, and makes it easy to upgrade everyone's software simultaneously. Additionally, because our staging and production environments are containerized, this approach ensures that the same piece of code will execute as predictably as possible across all of our environments.

#### dbt Command Reference

For standard dbt commands (`dbt run`, `dbt test`, `dbt compile`, `dbt seed`, etc.) see the [primary dbt command reference](https://docs.getdbt.com/reference/dbt-commands).

The [`transform/snowflake-dbt/Makefile`](https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/Makefile) wraps common development workflows. Run `make help` from within the venv shell for the full list. Key commands are:

| Command | Description |

|---|---|

| `make build-changes` | Clone and build all changed SQL/CSV files. Accepts `SELECTION`, `EXCLUDE`, `DOWNSTREAM`, `CONTIGUOUS`, `FAIL_FAST`, `FULL_REFRESH`, `VARS`, `TARGET` variables |

| `make lint-models` | Lint changed SQL files with SQLFluff. Accepts `MODEL` to target a specific file and `FIX=true` to apply fixes in place |

| `make safe-check` | Check for models downstream of SAFE data that are not properly tagged |

| `make row-counts` | Generate SQL queries to compare row counts between dev and prod (run after `make build-changes`) |

| `make list-models` | List models matching the current selection |

#### Build Changes Locally

To clone and build all of the changed models in the local development space the same `build_changes` process can be used that is used in the [CI Job](/handbook/enterprise-data/platform/ci-jobs/#build_changes).  The primary difference is that instead of a `WAREHOUSE` variable the developer can pass a `TARGET` variable to use a target configured with a different warehouse size.  To run the process, run the `make build-changes` command from within the virtual environment.

- Run `make dbt-deps` from the root of the analytics folder and retry the command.

**Transition Note:**

We are actively transitioning to the new `clone-dbt-select-local-user-noscript` command. The old `clone-dbt-select-local-user` command will still be available for a limited time, but we encourage you to start using the new command as soon as possible.

##### Cloning into local user DB (python scripts - pre-`dbt clone`)

- This clones the given dbt model lineage into the active branch DB (ie. `{user_name}_PROD`)

  - `make DBT_MODELS="<dbt_selector>" clone-dbt-select-local-user`

  - eg. `make DBT_MODELS="+dim_subscription" clone-dbt-select-local-user`

##### Cloning into branch DB

- This clones the given dbt model lineage into the active branch DB (ie. `{branch_name}_PROD`), this is equivalent to running the CI pipelines clone step.

- It does not work on Master.

  - `make DBT_MODELS="<dbt_selector>" clone-dbt-select-local-branch`

  - eg. `make DBT_MODELS="+dim_subscription" clone-dbt-select-local-branch`

### Production Orchestration

dbt models in production are orchestrated using [Apache Airflow](https://airflow.apache.org/). Each scheduled pipeline runs as an Airflow DAG inside a Kubernetes pod, executing dbt commands against Snowflake.

A model's eligibility for a given frequency is determined by its **full lineage build time** — not just the model's own runtime — to ensure the complete upstream graph can be built within the frequency window. Each model should appear in exactly one processing frequency to avoid duplicate builds.

### Docker Workflow {#docker-workflow}

The below is the recommended workflow primarily for users running Linux as the venv workflow has fewer prerequisites and is considerably faster.

To abstract away some of the complexity around handling `dbt` and its dependencies locally, the main [analytics project](https://gitlab.com/gitlab-data/analytics/) supports using `dbt` from within a `Docker` container.

We build the container from the [`data-image`](https://gitlab.com/gitlab-data/data-image) project.

There are commands within the `Makefile` to facilitate this, and if at any time you have questions about the various `make` commands and what they do, use `make help` to get a list of the commands and what each of them does.

Before your initial run (and whenever the containers get updated) make sure to run the following commands:

1. `make update-containers`

1. `make cleanup`

These commands will ensure you get the newest versions of the containers and generally clean up your local `Docker` environment.

#### Using dbt

- Ensure you have the `DBT_PROFILE_PATH` environment variable set. This should be set if you've used the [onboarding_script.zsh](https://gitlab.com/gitlab-data/analytics/-/blob/master/admin/onboarding_script.zsh) (recommened to use this as this latest and updated regularly) or   [onboarding_script.sh](https://gitlab.com/gitlab-data/analytics/blob/master/admin/onboarding_script.sh), but if not, you can set it in your `.bashrc` or `.zshrc` by adding `export DBT_PROFILE_PATH="/<your_root_dir/.dbt/"` to the file or simply running the same command in your local terminal session

- Ensure that you have updated your `.dbt/profiles.yml` with your specific user configuration

- Ensure your SSH configuration is setup according to the [GitLab directions](https://gitlab.com/help/ssh/README). Your keys should be in `~/.ssh/` and the keys should have been generated with no password.

  - You will also need access to [this project](https://gitlab.com/gitlab-data/data-tests) to run `dbt deps` for our main project.

- To start a `dbt` container and run commands from a shell inside of it, use `make dbt-image`

- This will automatically import everything `dbt` needs to run, including your local `profiles.yml` and repo files

  - You may see some WARNINGS about missing variables (`GIT_BRANCH`, `KUBECONFIG`, `GOOGLE_APPLICATION_CREDENTIALS`, etc.). Unless you are developing on Airflow this is ok and expected.

- To see the docs for your current branch, run `make dbt-docs` and then visit `localhost:8081` in a web-browser. Note that this requires the `docs` profile to be configured in your `profiles.yml`

- Once inside of the `dbt` container, run any `dbt` commands as you normally would

- Changes that are made to any files in the repo will automatically be updated within the container. There is no need to restart the container when you change a file through your editor!

#### Command line cheat sheet

This is a simplified version of the [primary command reference](https://docs.getdbt.com/reference/dbt-commands).

dbt specific:

- [`dbt clean`](https://docs.getdbt.com/reference/commands/clean) - this will remove the `/dbt_modules` (populated when you run deps) and `/target` folder (populated when models are run)

- [`dbt run`](https://docs.getdbt.com/reference/commands/run) - regular run

- Model selection syntax ([source](https://docs.getdbt.com/reference/node-selection/syntax)). Specifying models can save you a lot of time by only running/testing the models that you think are relevant. However, there is a risk that you'll forget to specify an important upstream dependency so it's a good idea to understand the syntax thoroughly:

  - `dbt run --models modelname` - will only run `modelname`

  - `dbt run --models +modelname` - will run `modelname` and all parents

  - `dbt run --models modelname+` - will run `modelname` and all children

  - `dbt run --models +modelname+` - will run `modelname`, and all parents and children

  - `dbt run --models @modelname` - will run `modelname`, all parents, all children, AND all parents of all children

  - `dbt run --exclude modelname` - will run all models except `modelname`

  - Note that all of these work with folder selection syntax too:

    - `dbt run --models folder` - will run all models in a folder

    - `dbt run --models folder.subfolder` - will run all models in the subfolder

    - `dbt run --models +folder.subfolder` - will run all models in the subfolder and all parents

- `dbt run --full-refresh` - will refresh incremental models

- [`dbt test`](https://docs.getdbt.com/reference/commands/test) - will run custom data tests and schema tests; TIP: `dbt test` takes the same `--model` and `--exclude` syntax referenced for `dbt run`

- [`dbt seed`](https://docs.getdbt.com/reference/commands/seed) - will load csv files specified in the `data-paths` [directory](https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt/data) into the data warehouse. Also see the [seeds section of this guide](/handbook/enterprise-data/platform/dbt-guide/#seeds)

- [`dbt compile`](https://docs.getdbt.com/reference/commands/compile) - compiles the templated code within the model(s) and outputs the result in the `target/` folder.

    This isn't a command you will need to run regularly as dbt will compile the models automatically when you to 'dbt run'.

    One common use-case is the compiled code can be run in Snowflake directly for debugging a model.

    Works only if you've run the [onboarding script](https://gitlab.com/gitlab-data/analytics/-/blob/master/admin/onboarding_script.sh):

- `dbt_run_changed` - a function we've added to your computer that only runs models that have changed (this is accessible from within the docker container)

- `cycle_logs` - a function we've added to your computer to clear out the dbt logs (not accessible from within the docker container)

- `make dbt-docs` - a command that will spin up a local container to serve you the `dbt` docs in a web-browser, found at `localhost:8081`

### VSCode extension: dbt Power User

[dbt Power User](https://marketplace.visualstudio.com/items?itemName=innoverio.vscode-dbt-power-user) makes VScode seamlessly work with dbt. The guide below will allow you to install dbt Power User if you followed the [Venv workflow](/handbook/enterprise-data/platform/dbt-guide/#Venv-workflow).

  "SNOWFLAKE_LOAD_DATABASE":"RAW",

  "SNOWFLAKE_STATIC_DATABASE":"STATIC",

  "SNOWFLAKE_PREP_SCHEMA":"preparation",

  "SNOWFLAKE_TRANSFORM_WAREHOUSE":"ANALYST_XS",

  "SNOWFLAKE_TRANSFORM_WAREHOUSE":"DEV_XS",

  "SALT":"pizza",

  "SALT_IP":"pie",

  "SALT_NAME":"pepperoni",

  When running/building/testing a model from VS code UI, the terminal window popping is only a log output. Cmd+C does not stop the job(s), nor clicking the Trash icon in VS code. If you want to stop a job started via VScode, go through the Snowflake UI and your job list and kill the job(s) from there.

  {{% /panel %}}

### Configuration for contributing to dbt project

If you're interested in contributing to dbt, here's our recommended way of setting up your local environment to make it easy.

- Fork the [dbt project](https://github.com/dbt-labs/dbt-core) via the GitHub UI to your personal namespace

- Clone the project locally

- Create a virtual environment (venv) for dbt following these commands

  ```bash

  cd ~

  mkdir .venv # This should be in your root "~" directory

  python -m venv .venv/dbt

  source ~/.venv/dbt/bin/activate

  pip install dbt

  ```

- Consider adding `alias dbt!="source ~/.venv/dbt/bin/activate"` to your `.bashrc` or `.zshrc` to make it easy to start the virtual environment

- Navigate to the dbt project in the same terminal window - you should see `(dbt)` at the start of the command prompt

- Run `pip install -r editable_requirements.txt`. This will ensure when you run dbt locally in your venv you're using the code on your machine.

- Run `which dbt` to ensure it's pointing to the venv

- Develop code locally, commit your changes as you would, and push up to your namespace on GitHub

When you're ready to submit your code for an MR, ensure you've [signed their CLA](https://github.com/dbt-labs/dbt-core/blob/dev/0.15.1/CONTRIBUTING.md#signing-the-cla).

## Style and Usage Guide

### Model Structure

Look at the [Use This Not That](https://docs.google.com/spreadsheets/d/1yr-J4ztkyl9vmJ6Euj58gczDLTIss7xIher5SV-1VDY/edit?usp=sharing) mapping to determine which new Kimball model replaces the legacy model.

{{% /panel %}}

{{% panel header="**FY21-Q4 Model Migration**" header-bg="success" %}}

In FY21-Q4 the `prod` and `prep` databases were introduced to replace the `analytics` database. These 2 new databases will fully replace the `analytics` database.

Local development was also switched from custom schemas to custom databases.

{{% /panel %}}

#### Sources

All raw data will still be in the `RAW` database in Snowflake.

}}

```

`product` and `non-product` models use a default warehouse size of `L`. If a model requires more compute — for example, a model with a large recursive CTE that times out — override the default at the model level using `generate_warehouse_name` as shown above. See the [Check Warehouse Size viability](/handbook/enterprise-data/platform/dbt-guide/#check-warehouse-size-viability) section below for guidance on when to increase size.

### Sample Data in Development

To streamline local development on local models, a way to sample, or use a subset of the data, is made available to the developers. This tool will allow developers the option of using sample data, or full data depending on what the situation calls for, allowing them to iterate quickly on the structure of models using sample data and then switch to full data when validation is needed. Using this in conjunction with local cloning of tables should improve the developer cycle time.

### Scheduling a dbt upgrade

dbt upgrades should take place on a Tuesday of a week in which there are no major worldwide holidays or [Family and Friends days](/handbook/company/family-and-friends-day/). This is to enable enough time for team members to correct any breaking changes that weren't caught in testing without having to work through the weekend. In a worst case scenario, the upgrade can be rolled back on Wednesday so that normal operations can resume for the remainder of the week.

## Specifying warehouse size on dbt model level

New `product` and `non-product` models will use a default 'L' warehouse size going forward.

Context: In order to decrease the runtime of the production dbt DAG, the `product` and `non-product` models have been consolidated into one Airflow task, [MR](https://gitlab.com/gitlab-data/analytics/-/merge_requests/11305). This means that the new default warehouse size will be 'L', and that if an 'XL' needs to be used, it needs to be specified on the dbt model level.

To specify the warehouse on the dbt model level (thus overriding the default warehouse), you need to add a config block to the model:

```sql

{{ config(

    snowflake_warehouse=generate_warehouse_name('XL')

) }}

```

[dim_note.sql](https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/models/common/dimensions_local/product_and_engineering/dim_note.sql) is an example model you can refer to.

For more info on choosing the correct warehouse, please see the 'Check Warehouse Size viability' section of this page.

you can use the [`dbt run`](https://docs.getdbt.com/reference/commands/run) command. Here

are some other useful resources and tips:

- [GitLab Data Team dbt command line cheat sheet](/handbook/enterprise-data/platform/dbt-guide/#command-line-cheat-sheet)

- [GitLab Data Team dbt command reference](/handbook/enterprise-data/platform/dbt-guide/#dbt-command-reference)

- [Model selection syntax overview](https://docs.getdbt.com/reference/node-selection/syntax)

- You can specify multiple models by including them in a command with a space in between the names

- You can specify multiple models by using [graph operators](https://docs.getdbt.com/reference/node-selection/graph-operators)