Favorita: Vertex AI KFP Pipelines
Chain the forecasting steps into a Kubeflow pipeline that runs on Vertex AI — compile a DAG of training, evaluation, and registration, then run it as one managed pipeline instead of a handful of disconnected jobs.
Primary Outcome
A compiled Vertex AI pipeline that runs the forecasting steps as one orchestrated, reproducible DAG.
Solution
A compiled Kubeflow pipeline that runs the full ML sequence as one parameterized, lineage-tracked Vertex AI job.
Deliverables
favorita_ml_pipeline.py— the KFP pipeline definitioncompile.py— compiles the pipeline to a runnable specA compiled pipeline artifact ready to submit to Vertex
Integration with the job submission layer
Strategic Context
Manual multi-step ML workflows are where reproducibility quietly dies — someone forgets a step or runs them out of order. Compiling the sequence into a pipeline makes the workflow itself a version-controlled artifact, which is the real point of pipeline tools.
Technical Architecture
Manual multi-step ML workflows are where reproducibility quietly dies — someone forgets a step or runs them out of order. Compiling the sequence into a pipeline makes the workflow itself a version-controlled artifact, which is the real point of pipeline tools.
Problem Statement
Running training, evaluation, and registration as separate manual steps invites skipped steps and irreproducible results.
Links
What's Included
The vertex/pipelines directory: the favorita_ml_pipeline.py KFP definition, compile.py to build the spec, and the compiled output, submitted via the Vertex jobs layer.
FAQs
KFP pipeline vs Prefect flow — which?
KFP orchestrates the ML steps inside Vertex; Prefect orchestrates the broader schedule including dbt. They complement each other.
Do I need Kubernetes knowledge?
No — Vertex Pipelines manages the infrastructure; you work at the KFP component level in Python.
Tech Stack
Tool 1
Tool 4
Tool 4
Tool 3
Tool 2
Tool 4
Primary Outcome
A compiled Vertex AI pipeline that runs the forecasting steps as one orchestrated, reproducible DAG.
Problem Statement
Running training, evaluation, and registration as separate manual steps invites skipped steps and irreproducible results.
Solution
A compiled Kubeflow pipeline that runs the full ML sequence as one parameterized, lineage-tracked Vertex AI job.
Links
Deliverables
favorita_ml_pipeline.py— the KFP pipeline definitioncompile.py— compiles the pipeline to a runnable specA compiled pipeline artifact ready to submit to Vertex
Integration with the job submission layer
What's Included
The vertex/pipelines directory: the favorita_ml_pipeline.py KFP definition, compile.py to build the spec, and the compiled output, submitted via the Vertex jobs layer.
Strategic Context
Manual multi-step ML workflows are where reproducibility quietly dies — someone forgets a step or runs them out of order. Compiling the sequence into a pipeline makes the workflow itself a version-controlled artifact, which is the real point of pipeline tools.
FAQs
KFP pipeline vs Prefect flow — which?
KFP orchestrates the ML steps inside Vertex; Prefect orchestrates the broader schedule including dbt. They complement each other.
Do I need Kubernetes knowledge?
No — Vertex Pipelines manages the infrastructure; you work at the KFP component level in Python.
Technical Architecture
Manual multi-step ML workflows are where reproducibility quietly dies — someone forgets a step or runs them out of order. Compiling the sequence into a pipeline makes the workflow itself a version-controlled artifact, which is the real point of pipeline tools.
Tech Stack
Tool 1
Tool 4
Tool 4
Tool 3
Tool 2
Tool 4