Standardize dbt Model Orchestration with Frequency-Based DAGs
## Overview With the company scorecard driving more frequent data updates, this epic formalizes and standardizes how dbt models are assigned to Airflow DAGs based on update frequency. Models will declare a `config.meta.frequency` value, dbt selectors will be built from those configs, and Airflow DAGs will be driven by those selectors — providing transparent, manageable, and standardized control over data refresh cadence. ## Supported Frequencies - `one_hour` - `four_hour` - `six_hour` - `eight_hour` - `twelve_hour` - `twenty_four_hour` - `weekly` - `monthly` - `first_week_of_month` ## Key Design Principles - The default assumption is daily processing (`twenty_four_hour`) - Monthly maintenance processing takes precedence over all other frequencies - Full lineage build time determines eligibility for a given frequency - Each model should appear in only one processing frequency to avoid overlap - Standardize on using `dbt build` command and executing all nodes but tests together. ## Child Issues - gitlab-data/analytics#26719 Handbook: Document the dbt Processing System Standard - gitlab-data/analytics#26720 Implement `config.meta.frequency` model configuration standard - gitlab-data/analytics#26721 Build frequency-based selectors in `selectors.yml` - gitlab-data/analytics#26722 Build frequency-based DAG entries in `dbt_jobs_manifest.yml` - gitlab-data/analytics#26723 Audit and migrate existing models to assigned processing frequencies
epic