When Agile Meets Machine Learning
5 Guiding Principles in Implementing Agile Methodology in Research-Intensive Software Environments
Agile software development is becoming the common practice in more and more software organizations. For those organizations that use to work in the traditional, waterfall way, going agile means letting many myths go. It’s never easy, however, when dealing with pure functional products, it is often the case that within a reasonable period of time, the superiority of the new methodology becomes apparent, and that often wins the argument.
In addition to the regular and inherent pain that is caused by the change, implementing agile in domains that are research intensive (for example, with products that are based on machine learning) has some additional obstacles. Research contains significantly higher degree of uncertainty. Research means finding new knowledge, that is currently unavailable, and predicting the exact form of that knowledge, and the resources that are needed to find it is hard. Moreover, researchers are trained and educated in academy, where in many cases there are no customers to collaborate with, comprehensive documentation is the ultimate outcome, and the tightest compliance with tools and standards is required, in order to compare with previous works. Not surprisingly, the term “research” is intimidating to many managers and business personnel.
“Does agile methodology fit in research intensive software environments?”
The question above seems like a rhetoric question, given the title of this post, but it is being asked by many many software organizations, including organizations that run agile fluently, in their pure functional features. Undoubtedly, the answer is a big yes; however, research intensive software environments has some unique characteristics that need to be considered during almost every aspects of agile implementation.
In this post we suggest 5 key considerations, that deserve closer attention, when implementing agile methodology in research-intensive software environments.
1. It is not all or nothing
In pure functional features, there are two important considerations: correctness and engineering quality. The team will typically not commit to a feature if it cannot be supplied correctly and with sufficient quality. In functional features correctness is usually well defined.
In research intensive features, the knowledge that is required for developing the new feature is initially absent, and so is often the exact outcome. In many cases, there may be several levels of correctness.
Consider for example an analytic feature that includes a prediction model. With no model, the prediction is random. With the “optimal” model, the prediction accuracy might be, for example, 95%. However, the concept of an optimal model is merely theoretical, since in reality we cannot asses the accuracy of an optimal model, and getting near this 95% requires a lot of efforts. It is often the case that really quickly a skillful data scientist can come with a model that can provide 85% of accuracy, or even 90% of accuracy. Assuming that this model is implemented with high engineering quality, the question of correctness remains: did we fully optimized the solution already or not?
Going agile requires many times to prefer quick solutions over optimal ones. Having a solution that is already providing significant business value, it is often the case that we better start implementing the model, instead of continue optimizing it.
As a matter of fact, prediction tasks are an easy example, in the sense that the solutions have objective measures of correctness (the prediction accuracy in this case). How would you measure or compare algorithms that cluster objects into groups, or algorithms that automatically summarize long documents? Research domains often require baby steps, and solutions that improve iteratively. Research is a product that evolves continuously and not a one-time project. Giving up the aim for optimization is sometimes hard for researcher who were educated in academy, where user engagement is no consideration, and it often requires a cultural change.
2. Create a buffer to reduce the impact of uncertainty
Uncertainty regarding the outcomes is common to any significant research. It is true that any planning includes some uncertainty, but in research, the level of uncertainty is by far higher. This huge uncertainty makes it dangerous to include the research and the implementation of the same feature, in a single iteration or plan. In research, failing to prove feasibility is an acceptable outcome (finding another way that is not working is frustrating, but it happens a lot). Failing to deliver a feature is less acceptable. Counting on the research phase to result in a feasible solution, in order to immediately implement this solution, by definition means that many times you will end with undelivered feature.
In most cases you even have to maintain a small buffer of research-oriented features that are waiting for implementation. Without this buffer, the implementation team will eventually starve to inputs, and be underutilized. Having a too big buffer is not recommended either, because it disturbs the important connection between the research and the customers’ needs and makes production too slow.
3. Build heterogeneous teams
Diversity and heterogeneity are blessed, in many cases. Unfortunately, many organizations see their researchers as individual contributors. Researchers are viewed as highly smart individuals, yet strange and sometimes hard to work with. Individual-contributor researchers get very little feedback from customers, very little feedback from developers and very little feedback from product owners. No wonder that in some cases individual contributors indeed lose their connections with reality, and may provide outcomes that do not take into account all the product and infrastructure constraints.
Not less important, in the individual contributor mode, customers, product owners and developers get very little feedback from the researchers, and often tempted to think that research is some sort of magic. When this is the case, the product comes with unrealistic request and expectations, and fail to deliver these requests to the researchers, and developers fail to prepare the infrastructures for implementing the new features.
Teams that conduct research should not be comprised solely from researchers. The developers who will implement the feature under research and the product representative should be part of the team, as well.
Moreover, cross-role knowledge sharing should be encouraged. Researchers are often highly curious, open minded individuals, that can contribute in many domains, and sharing business knowledge and development knowledge with them, creates a win-win. Research work is often highly interesting, and sharing it with product and development is also a win-win, since it increases the level of work interest, and helps in aligning expectations.
4. Continuous improvement (retrospective)
Retrospective is one of the strongest aspects of agile. Dedicating time to discuss the way that the work is done, rather than the work itself, and adapting the methodology according to the team’s needs and desires ultimately drive teams to become more engaged and self-empowered, and simply makes the work methods better. With heterogeneous teams (see guiding principle no. 3) agile become even more complicated and comes with higher degrees of freedom. The different terminology, mode of work, outcomes and even mentality, should be discussed and learned by all sides with much respect and patience to each other. The more diverse is the team, the more important it becomes to have deep non-judgmental retrospective. Going agile in research intensive environments takes way more time and improvements iterations, and missing the needed room for retrospective jeopardize the level of team satisfaction from the change, the probability of finding some sweet implementation method and eventually the overall success of the change.
5. Its all about the business
Differently from basic research, in which the sole objective is generating new knowledge, and where wondering about the value of that new knowledge is sometimes non-legitimate, in business domain research should be applicative, and serve the business objectives. Now, when written, it hard to argue that every applicative research should stem from a business need and serve that business need. Nonetheless, there are many efforts, in so many organizations, which are spent on research from no clear reason. This phenomenon especially characterizes emerging and sexy domains (like big data and machine learning, in the recent years).
Conducting research with no clear business need is not only an almost certain recipe for failure, it is also a stick in the wheels of agile. With no clear business needs there is no clear way to define iterations outcome, no way to commit on anything and therefore no way to proceed. The most that can come out from such non-applicative research is a sudden enlightenment. I know very few organizations that can afford themselves the expectation of arbitrary enlightenment as a business model.
