What is dbt in data engineering? Architecture guide

Here's what dbt actually does, where it fits in an ELT stack, and how to judge if it belongs in yours.

What is dbt in data engineering?

In an ELT pipeline, raw data lands in the warehouse first; dbt then compiles SQL and Jinja logic into a directed acyclic graph (DAG) of models that the warehouse runs natively.

Extract and load stay with tools like Fivetran; dbt owns transform, and teams use it to turn ad hoc SQL code into version-controlled models they can read, test, and reuse.

If you're weighing whether to build this transformation layer in-house or need expert data engineering support to get it right, the setup decisions compound quickly.

dbt started in 2016 as an open-source project at Fishtown Analytics, the company that later became dbt Labs, and it has since become the default transform layer for ELT pipelines running on modern cloud warehouses.

The practical gain for teams moving off legacy stored procedures is visibility: transformation logic sits in one model layer instead of scattered procedures, and data lineage becomes traceable end to end instead of buried in undocumented jobs.

If you'd rather bring in outside expertise for this kind of migration, our guide to vetting a data engineering partner walks through selection criteria and top vendors.

Where does dbt sit in an ELT pipeline?

dbt sits at the transform stage of ELT (extract, load, transform), after raw data lands in the warehouse, before it reaches BI tools or reverse-ETL jobs. It never touches extraction or load; that boundary is the most commonly misunderstood part of the setup.

Fivetran (or Airbyte, Stitch) pulls records from source systems and lands them untouched in Snowflake, BigQuery, or Redshift. dbt takes over from there, compiling SQL and Jinja logic into a DAG the warehouse runs as models, promoting raw tables through staging, intermediate, and marts layers teams can read and reuse.

Airflow (or Dagster, Prefect) sits above both, scheduling the load job, then triggering the dbt run once new data lands. dbt doesn't schedule anything on its own; dbt Cloud and the Fusion engine add native scheduling, but Airflow remains the standard orchestrator for teams already running multi-tool pipelines.

The two sides now sit under one company: Fivetran and dbt Labs announced their merger in October 2025 and completed it on June 1, 2026. dbt Core remains open source, and extract-load and transform are still separate steps in the pipeline.

In practice, the pattern looks like this: Airflow triggers the Fivetran sync, then calls dbt build against the same warehouse, so transformation runs as a dependency of the load instead of on a separate cron schedule that can fire before the data arrives.

Get the layering wrong and lineage breaks. dbt can read Fivetran's raw tables but can't detect a silently failed load job, which is why the two need separate monitoring, not one shared alert.

What is dbt used for? Example workflow

dbt is the layer that turns raw tables sitting in a data warehouse into clean, tested models an analytics engineer can query without re-deriving business logic every time. The standard structure is staging-to-marts model layering: staging models cast raw loaded source tables into typed, renamed columns; intermediate models join and dedupe; mart models aggregate everything into the tables BI tools actually read.

A typical run looks like this: a source table lands in Snowflake or BigQuery, a staging model wraps it in a ref call, an intermediate model joins three staging models on a shared key, and a mart model rolls the result into a revenue-by-cohort table.

dbt run compiles the Jinja and SQL into warehouse-native code and executes it in DAG order, so a model never runs before its upstream dependency.

Schema and data tests run inline with each build, so a broken foreign key or an unexpected null gets caught in CI before it reaches a mart, not after a dashboard breaks.

That matters because bad data reaching stakeholders is the top worry in the field: in dbt Labs' 2026 State of Analytics Engineering report, 71% of data professionals said they were concerned about it, and trust in data was the most widely prioritized goal, at 83% (PR Newswire).

Whether you run dbt Core, the dbt platform, or the newer Fusion engine changes job monitoring, observability depth, and cost per run, but the staging-to-marts logic underneath stays identical across all three.

What is a dbt model?

A dbt model is a single .sql file containing a SELECT statement, one node in the directed acyclic graph (DAG) that dbt compiles and runs in dependency order. There is no INSERT or CREATE TABLE in the file itself. dbt's Jinja-to-SQL compiler handles that, expanding ref calls, macros, and config blocks into warehouse-native SQL before handing it to the runner.

The ref function is what makes the DAG possible. Instead of hardcoding a downstream table name, a model calls ref('stg_orders'), and dbt resolves that to the correct schema, tracks it as a dependency edge, and generates the lineage graph you see in dbt docs. Break a ref, and the DAG won't compile, catching a broken pipeline before it hits production.

Materialization decides how a model lands in the data warehouse: view computes on every read, table rebuilds fully each run, incremental appends or merges only new rows. Moving a large fact model from table to incremental usually gives the biggest runtime drop in a project, because the warehouse stops rescanning history it has already processed.

Here is what a small incremental model looks like in practice:

-- models/marts/fct_orders.sql
{{ config(materialized='incremental', unique_key='order_id') }}

select
    o.order_id,
    o.customer_id,
    o.ordered_at,
    sum(i.amount) as order_total
from {{ ref('stg_orders') }} as o
join {{ ref('stg_order_items') }} as i
    on o.order_id = i.order_id
{% if is_incremental() %}
where o.ordered_at > (select max(ordered_at) from {{ this }})
{% endif %}
group by 1, 2, 3

The two ref() calls create the DAG edges, config() sets the materialization, and the is_incremental() block limits each run to new orders. dbt compiles all of it into plain SQL for your warehouse.

Good incremental models stay idempotent: rerunning them after a partial failure produces the same result set, not duplicates, which is why the is_incremental macro and a reliable unique_key matter more than most teams initially assume.

How dbt compiles and runs SQL (Jinja mechanics)

dbt separates compilation from execution: a compiler resolves Jinja templating and macros into raw SQL, then a runner sends that SQL to the warehouse for execution. Understanding this two-step process explains why dbt code looks nothing like what actually runs.

When you invoke dbt run, dbt first parses every model, expanding ref calls, {% for %} loops, and macros into warehouse-native SQL. Macros work like reusable functions: write a currency-conversion macro once, call it across twenty models, and a logic change propagates everywhere on the next build. This is the same DAG that determines execution order, so compilation and dependency resolution happen in one pass.

Only after compilation does dbt hand off SQL for execution against Snowflake, BigQuery, or another warehouse, tracking compute independently of the compile step. This split matters for cost: compiling is nearly free CPU work, but every executed model consumes warehouse compute, so bloated macros or unnecessary full-refreshes hit your bill, not your build logs.

dbt Labs' Fusion engine, launched in May 2025 and built on technology from its January 2025 acquisition of SDF Labs, rewrites this engine in Rust for faster parsing on large projects, a meaningful shift for teams running thousands of models where dbt Core's Python-based compilation becomes the bottleneck.

How dbt builds data lineage with the DAG

dbt builds its directed acyclic graph (DAG) automatically from every ref call in your model code, mapping which tables read from which upstream models before a single query runs. There's no separate lineage tool to maintain: the dependency graph is a byproduct of writing SQL the dbt way.

Each time a model references another with ref('stg_orders') instead of a hardcoded table name, dbt records an edge in the graph.

Compile the project and dbt (Core or Fusion) resolves the full DAG, orders execution so upstream models finish first, and blocks circular references outright, hence acyclic.

This is why staging-to-marts layering works cleanly: marts models never build before the staging models they depend on.

The payoff shows up in production. When one staging model breaks, the DAG lets you rerun just that model and its downstream marts (for example with dbt build --select stg_orders+) instead of the entire nightly job.

Teams also get a visual lineage graph in the dbt platform and the open-source dbt docs site, which turns that dependency graph into something a non-engineer can read.

dbt Core vs dbt Cloud vs Fusion engine

dbt Core, dbt Cloud (renamed the dbt platform in 2025), and the newer Fusion engine split on where compilation and orchestration run, and that split drives your total cost of ownership more than any list price does.

The Fusion engine replaces dbt Core's Python-based engine with one written in Rust, built for faster parsing across large DAGs. It predates the Fivetran merger: it came out of dbt Labs' SDF Labs acquisition.

The gap that actually matters usually isn't Core versus Cloud pricing, it's whether your team wants to run its own job monitoring and lineage tooling or buy it.

dbt Core keeps compute cost lean but pushes scheduling, alerting, and access control onto your platform team's existing stack. dbt Cloud folds job monitoring, run history, and exposure tracking into the product, which shows up as a line item, not a warehouse bill.

For teams standardizing staging-to-marts model layering across dozens of pipelines, that operational overhead is the real comparison to run, not the sticker price.

How dbt handles testing and documentation

dbt treats schema and data tests as compiled SQL that runs against your warehouse on every build, failing the pipeline before bad data reaches a dashboard. A schema test checks structure: not_null, unique, relationships between staging and marts models. A data test is arbitrary SQL asserting a business rule, like revenue never going negative, which helps maintain data quality across your analytics pipeline.

Both compile through the same Jinja-to-SQL runner as your models, so tests live in version control next to the transformation logic they guard, not in a separate QA tool.

A relationships test on an orders-to-customers join is the classic example: it catches orphaned foreign keys that a procedural pipeline would pass through silently.

Documentation is generated the same way. Running dbt docs generate reads your YAML descriptions and model SQL, then builds a searchable site with a model-level lineage graph across the full DAG, so anyone can trace a metric back to its source tables without asking the person who wrote it.

dbt Labs' own documentation recommends layering generic tests (schema-level) with singular tests (custom SQL) rather than relying on one type.

dbt with Snowflake, BigQuery, and the package manager

dbt connects to Snowflake and BigQuery through adapters that compile the same Jinja-templated SQL into each warehouse's native dialect, so a model written once runs unmodified against either engine. This matters for teams running multi-warehouse setups after an acquisition or a phased warehouse migration, the transformation logic doesn't need a rewrite, only an adapter swap.

Warehouse choice does change compute behavior worth watching. Snowflake bills by virtual warehouse runtime, so an incremental model with a poorly scoped is_incremental filter can scan far more data than a full refresh. BigQuery's slot-based billing rewards similar discipline through partition pruning in the same filter clause.

dbt's biggest practical advantage over competing transformation tools is its package ecosystem. The dbt Package Hub distributes reusable macros and models, dbt_utils for surrogate keys and date spines, dbt_expectations for statistical tests, that a team installs with one dbt deps command rather than writing from scratch. Packages are declared in a packages.yml file at the project root:

packages:
  - package: dbt-labs/dbt_utils
    version: [">=1.0.0", "<2.0.0"]

Pinning a version range keeps upgrades deliberate instead of accidental. This is where the open-source community, not just dbt Labs, extends the DAG's logic across warehouses.

dbt vs Airflow vs Dagster: Do you need both?

dbt handles transformation logic, not scheduling. Airflow and Dagster sit one layer up, triggering the dbt run (or dbt build) command alongside ingestion jobs, reverse-ETL syncs, and downstream analytics tasks. Most teams running dbt in production need one of them.

The distinction shows up in the DAG itself. dbt's DAG maps lineage between SQL models inside the warehouse, model A feeds model B via ref. Airflow's DAG maps task dependencies across systems: a Fivetran load, then a dbt run, then a Slack alert.

Dagster goes further, treating dbt models as native software-defined assets, so its lineage view merges orchestration and transformation into one graph instead of two separate ones teams have to reconcile manually.

Cost matters here too: dbt Core plus a self-hosted orchestrator is cheaper to run but costs engineering time; the managed dbt platform trades that time for a subscription line.

FAQ: dbt in data engineering