Favorita: Data Ingestion to BigQuery
The on-ramp: scripts that load the raw Favorita data into BigQuery and apply the Vertex-related table DDL. Before dbt can transform anything, the data has to land — this is the step everyone skips in tutorials.
Primary Outcome
A repeatable way to load raw data into BigQuery and create the tables the rest of the project depends on.
Solution
Scripts that load the raw data into BigQuery and apply the required DDL, making the data layer reproducible from zero.
Deliverables
load_favorita_to_bigquery.py— loads the raw dataset into BigQueryapply_vertex_bq_ddl.py— creates the Vertex-related tables via DDLRepeatable setup so a fresh project can be stood up from scratch
Strategic Context
Ingestion is the step demos pretend doesn't exist, and the step real projects spend the first week on. Scripting it makes the difference between a project anyone can reproduce and one that only works on the original author's machine.
Technical Architecture
Ingestion is the step demos pretend doesn't exist, and the step real projects spend the first week on. Scripting it makes the difference between a project anyone can reproduce and one that only works on the original author's machine.
Problem Statement
Tutorials assume the data is already in the warehouse, leaving the ingestion step undocumented and projects impossible to reproduce from scratch.
Links
What's Included
The scripts directory: load_favorita_to_bigquery.py for loading the raw data and apply_vertex_bq_ddl.py for applying the table DDL the Vertex layer needs.
FAQs
Do I need the Favorita data first?
Yes — download the Kaggle dataset, then point the load script at it. The DDL script needs only your GCP project.
Is this safe to rerun?
It's built to bootstrap or rebuild the data layer; review the scripts before rerunning against a populated project.
Tech Stack
Tool 1
Tool 4
Tool 4
Tool 3
Tool 2
Tool 4
Primary Outcome
A repeatable way to load raw data into BigQuery and create the tables the rest of the project depends on.
Problem Statement
Tutorials assume the data is already in the warehouse, leaving the ingestion step undocumented and projects impossible to reproduce from scratch.
Solution
Scripts that load the raw data into BigQuery and apply the required DDL, making the data layer reproducible from zero.
Links
Deliverables
load_favorita_to_bigquery.py— loads the raw dataset into BigQueryapply_vertex_bq_ddl.py— creates the Vertex-related tables via DDLRepeatable setup so a fresh project can be stood up from scratch
What's Included
The scripts directory: load_favorita_to_bigquery.py for loading the raw data and apply_vertex_bq_ddl.py for applying the table DDL the Vertex layer needs.
Strategic Context
Ingestion is the step demos pretend doesn't exist, and the step real projects spend the first week on. Scripting it makes the difference between a project anyone can reproduce and one that only works on the original author's machine.
FAQs
Do I need the Favorita data first?
Yes — download the Kaggle dataset, then point the load script at it. The DDL script needs only your GCP project.
Is this safe to rerun?
It's built to bootstrap or rebuild the data layer; review the scripts before rerunning against a populated project.
Technical Architecture
Ingestion is the step demos pretend doesn't exist, and the step real projects spend the first week on. Scripting it makes the difference between a project anyone can reproduce and one that only works on the original author's machine.
Tech Stack
Tool 1
Tool 4
Tool 4
Tool 3
Tool 2
Tool 4