Standardize dbt Model Orchestration with Frequency-Based DAGs
## Overview
With the company scorecard driving more frequent data updates, this epic formalizes and standardizes how dbt models are assigned to Airflow DAGs based on update frequency.
Models will declare a `config.meta.frequency` value, dbt selectors will be built from those configs, and Airflow DAGs will be driven by those selectors — providing transparent, manageable, and standardized control over data refresh cadence.
## Supported Frequencies
- `one_hour`
- `four_hour`
- `six_hour`
- `eight_hour`
- `twelve_hour`
- `twenty_four_hour`
- `weekly`
- `monthly`
- `first_week_of_month`
## Key Design Principles
- The default assumption is daily processing (`twenty_four_hour`)
- Monthly maintenance processing takes precedence over all other frequencies
- Full lineage build time determines eligibility for a given frequency
- Each model should appear in only one processing frequency to avoid overlap
- Standardize on using `dbt build` command and executing all nodes but tests together.
## Child Issues
- gitlab-data/analytics#26719 Handbook: Document the dbt Processing System Standard
- gitlab-data/analytics#26720 Implement `config.meta.frequency` model configuration standard
- gitlab-data/analytics#26721 Build frequency-based selectors in `selectors.yml`
- gitlab-data/analytics#26722 Build frequency-based DAG entries in `dbt_jobs_manifest.yml`
- gitlab-data/analytics#26723 Audit and migrate existing models to assigned processing frequencies
epic