Business Science
EvoFlow: why we preferred our own solution to Airflow
A versatile flow of flow checks
An introduction to EvoFlow
EvoFlow is a versatile platform developped by Evo’s data engineer and devOps teams that creates, schedules and monitors workflows.
Codifying workflows allows the development teams to create and share versions of their products. This collaborative process can all happen while maintaining the structure of those products or, even better, while improving those structures.
EvoFlow was created to be an enhanced flow of ETL, Scraper, and other data pipeline checks, but its functionalities expand far beyond these uses.
The EVO way
So why did we create our own platform? After all, the widely-adopted, AirFlow (https://airflow.apache.org/) can do all of the things that EvoFlow does; it has long become the standard. Although the usefulness of AirFlow is not in dispute, the tool can be somewhat unwieldy. With so much functionality, it can actually slow down simple checks. We needed a tool to simplify ETL and Scraper flows, not to complicate the process further.
The EVO data engineer and DevOps teams decided to build a simplified platform that would accomplish…