Favorita: Prefect Orchestration Flows
The orchestration layer that ties it together: Prefect flows and tasks that run the dbt build and the Vertex pipeline on a schedule, with the shared config and repo utilities they rely on. The conductor for the whole stack.
Primary Outcome
Scheduled, observable Prefect flows that run the dbt and Vertex steps as one coordinated workflow.
Solution
Prefect flows and tasks that schedule and coordinate the dbt build and Vertex training as one observable workflow.
Deliverables
flows/dbt.py— a flow that runs the dbt buildflows/vertex.pyandflows/vertex_pipeline.py— flows for Vertex training and pipelinestasks/— reusable dbt and Vertex tasksutils/— shared config, pipeline, and repo helpersA
prefect.yamldeployment definition at the repo root
Strategic Context
A forecasting stack is only as reliable as its orchestration. Transforms and retraining that drift out of sync produce forecasts on stale features — a subtle, expensive failure. Putting both under one scheduler is what turns a pile of scripts into a system.
Technical Architecture
A forecasting stack is only as reliable as its orchestration. Transforms and retraining that drift out of sync produce forecasts on stale features — a subtle, expensive failure. Putting both under one scheduler is what turns a pile of scripts into a system.
Problem Statement
Transforms and model retraining run as separate manual steps that drift out of sync, producing forecasts on stale features.
Links
What's Included
The orchestration package: flows/ (dbt, vertex, vertex_pipeline), tasks/ (dbt, vertex), and utils/ (configs, pipelines, repo), plus the root prefect.yaml deployment file.
FAQs
Why Prefect over Airflow here?
It's Python-native and light to stand up; the flow/task split maps cleanly onto dbt and Vertex steps. The pattern ports to Airflow or Dagster.
Does this replace the KFP pipeline?
No — KFP orchestrates inside Vertex; Prefect orchestrates the broader schedule. They work together.
Tech Stack
Tool 1
Tool 4
Tool 4
Tool 3
Tool 2
Tool 4
Primary Outcome
Scheduled, observable Prefect flows that run the dbt and Vertex steps as one coordinated workflow.
Problem Statement
Transforms and model retraining run as separate manual steps that drift out of sync, producing forecasts on stale features.
Solution
Prefect flows and tasks that schedule and coordinate the dbt build and Vertex training as one observable workflow.
Links
Deliverables
flows/dbt.py— a flow that runs the dbt buildflows/vertex.pyandflows/vertex_pipeline.py— flows for Vertex training and pipelinestasks/— reusable dbt and Vertex tasksutils/— shared config, pipeline, and repo helpersA
prefect.yamldeployment definition at the repo root
What's Included
The orchestration package: flows/ (dbt, vertex, vertex_pipeline), tasks/ (dbt, vertex), and utils/ (configs, pipelines, repo), plus the root prefect.yaml deployment file.
Strategic Context
A forecasting stack is only as reliable as its orchestration. Transforms and retraining that drift out of sync produce forecasts on stale features — a subtle, expensive failure. Putting both under one scheduler is what turns a pile of scripts into a system.
FAQs
Why Prefect over Airflow here?
It's Python-native and light to stand up; the flow/task split maps cleanly onto dbt and Vertex steps. The pattern ports to Airflow or Dagster.
Does this replace the KFP pipeline?
No — KFP orchestrates inside Vertex; Prefect orchestrates the broader schedule. They work together.
Technical Architecture
A forecasting stack is only as reliable as its orchestration. Transforms and retraining that drift out of sync produce forecasts on stale features — a subtle, expensive failure. Putting both under one scheduler is what turns a pile of scripts into a system.
Tech Stack
Tool 1
Tool 4
Tool 4
Tool 3
Tool 2
Tool 4