End to End Data Engineering Project using Databricks Free Edition | Spark Declarative Pipelines

The tutorial focuses on building a complete, end-to-end data engineering project utilizing the Databricks Free Edition and Databricks Lakeflow Spark Declarative Pipelines (SDP). The project is situated within the transportation domain, demonstrating practical application of these technologies for resume-building purposes.

The content covers the foundational steps, beginning with project introduction, stakeholder discussion, and an overview of the technical architecture and data understanding. Key technical segments include establishing the Databricks environment, setting up catalogs and schemas, and connecting to external storage like S3 buckets. A comparison between declarative and imperative programming is also provided.

The practical implementation involves staging data through medallion architecture layers: Bronze tables (e.g., City, Trips), Silver tables (providing dimensional clarity), and culminating in the Gold layer for business intelligence (BI) ready data. The session emphasizes the concepts underpinning Spark Declarative Pipelines throughout the ETL/ELT process.