Integrating Prefect & Databricks to Manage your Spark Jobs

Photo taken by Justin Jairam from @jusspreme (with permission)

Prefect is a workflow management system that enables users to easily take data applications and add retries, logging, dynamic mapping, caching, failure notifications, scheduling and more — all with functional Python API. Prefect allows users to take their existing code and transform it into a DAG (Directed Acyclic Graph) with dependencies already identified [1]. It simplifies the creation…