Within a Big Data project, topics such as building a Data Warehouse or Datalake, data integration, implementation of a BI tool or an AI/DL model often occur. Several project management approaches and tools are available for the course of a project. In this relation, one differentiates mostly between these three methods: agile, classic and hybrid.
In the past, projects like the creation of data warehouses was a large, monolithic, quarterly or multi-year effort, and subject to the traditional "waterfall" process. In the modern era, this is no longer the norm, with many organizations opting to use a more flexible and iterative or agile approach. With business needs changing faster than ever and new data sources being added all the time, the solution is usually to adopt an incremental and agile manner. With an iterative approach, tasks/sub-projects can thus be broken down into smaller stories and business value can be created faster. As a result, the following three approaches have emerged and become the most common methods for implementing a data science project[1]:
Method 1: Scrum
This approach is the most widely used process framework for agile development processes. Scrum emphasizes daily communication and flexible reassessment of plans that are executed in a short period of time. The agile Data Science Manifesto is organized around the following principles [2]:
- Iteration
- Ship intermediate output
- Prototype experiments over implementing tasks
- Innovative product management
- Strong customer focus
Method 2: Kanban
This approach features managing/improving products with a focus on continuous delivery without overloading the development team.

Like Scrum, Kanban is a process to help teams work together by visualizing workflows, WIP boundaries, workflow management, and creating guidelines, it aims to improve collaboration and process.
Method 3: BEAM
Beam aims at Agile Dimensional Modeling, with the goal of aligning requirements analysis with business processes rather than reports. The basic principles of this approach are [3][4]:
- Individuals and Interactions: Business intelligence is driven by what users ask about their business. The technical setting is secondary.
- Business Driven: Well documented data warehouses that take years to deploy will always be out of date. Business users will look elsewhere. My experiences with business units: I need it now or I’d rather stick with my familiar Excel solution ….
- Customer Collaboration: The end users’ knowledge of their business is your greatest resource.
- Responding to Change: If you take all of the above actions, change will come naturally and results in weekly delivery cycles.
The author of BEAM, Lawrence Corr, describes, among other things, a design pattern for structuring of "data stories". Example: Business Event – Customer buys product [5]:
- When was the order date ?
- Where was the product purchased, where will it be delivered?
- How many quantities were bought?
- What was the reason behind the purchase?
- Which channel was used for the purchase of the product?
Through these important questions the business process comes to light and so technical conditions can be derived.
Conclusion
In this article, the three most commonly known project management approaches were mentioned and further described. However, it is important to realize that in addition to finding the best methodology, understanding that a shift to the agile methodology or automated code development is not just a shift in skills. It is rather a shift in mindset. In the field of Data Science, agile approaches are promising because it is often impossible to assess in advance, whether the project can be achieved at all using the available data. The extent to which the results are successful can only be judged once they are available.
Source and Further Readings
[1] Jeffrey S. Saltz, Ivan Shamshurin, Kevin Crowston, Comparing Data Science Project Management Methodologies via a Controlled Experiment (2017), Proceedings of the 50th Hawaii International Conference on System Sciences
[2] Russell Jurney, Agile Data Science 2.0
[3] optimalbi.com, AgileBI vs BEAM✲ vs Modelstorming (2015)
[4] Lawrence Corr, Jim Stagnitto, Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema (2011)
[5] Raphael Branger, BI-Excellence durch Agilität und Automatisierung (2017)