The world’s leading publication for data science, AI, and ML professionals.

Build The Right Machine Learning Team, Not The Best

Learn the pitfalls in building a team that delivers advanced analytics capabilities through machine learning

Advanced Analytics in Business Problem Solving. Figure by Photomix.
Advanced Analytics in Business Problem Solving. Figure by Photomix.

Advanced analytics powered by Machine Learning (ML) are all in rage. Companies, large or small, are bringing up forecasting, recommending, classifying, etc., more than ever and creating teams that are tackling these concepts in different problem areas.

A key question that needs to be answered is how to build such a team. Often, the goal of the team is developing and/or managing a software system and a process around the system. At their core, they manage data, code, model, and insight. I have seen that teams are jumping to solve hyper-specialized problems at every opportunity. Thus, they are requesting multiple data scientists, machine learning engineers, data analysts, etc., to join the team as soon as possible. Is this the right way to tackle an unknown problem and build a team?

We live in a landscape where no two challenges are the same. For two grocery chains operating in the same country and trying to find the right discount prices for their products may not be the same challenge. One grocery channel may attract affluent customers whereas the other one may be more for bargain hunters. For the same grocery, experience in the optimized ordering of long-shelf products may not translate easily to seasonal products. It may be tempting to think that skills, such as experience in retail, data analysis, algorithms, data and modeling pipelines, etc., are sufficient. In reality, having such professionals are an advantage but not sufficient. Delivering value goes beyond skill to do technical work. It is important to figure out how to tie these experiences in the right way at the right time.

Personally, I feel that I am somewhere in the early phase of my learning how to do it. However, I have learned a few things already that I want to share with the community. In this article, I will explain the stages that a team goes through in delivering advanced analytics. Each of these stages comes with unique requirements that the team needs to address to deliver the desired results. Addressing the requirements correctly is at the heart of building the right team.

Stages of machine learning team

Figure 1: Stages of an ML product team. Figure by the author.
Figure 1: Stages of an ML product team. Figure by the author.

Figure 1 presents the five main stages of an ML product team. We describe the stages and the skills requirements in the following.

  • Exploring: In this stage, the team needs to isolate the key problems to solve. At the same time, the team needs to figure out the data that could provide more insight in approximating the problem scenario, which would inspire the solution design. In this stage, it’s important to scan and stay broad than to go deep. Moreover, it is crucial to communicate the findings and follow-up ideas among the team daily and with the stakeholder regularly but less frequently. The outcome of this stage is that the team has come up with some data-driven formulation of the identified problems. Skills required at this stage include knowledge in the problem area, knowledge in the data relevant to the problem area, data analysis, and visualization, requirement analysis, design thinking, communication, etc. Note that some of these skills are rather entrepreneurial skills that would be applied in a startup environment, which may come from the product owner or project manager as well as data or business analysts.
  • Testing: In this stage, the team should focus on implementing the solution of the formulated problem that allows them to test a few, but key hypotheses. Obviously, the test development should go parallel to solution development so that mistakes can be immediately identified. Once the solution works in a test environment, it can be tested in a target market on a limited scope. Over time, the scope can be broadened in phases. Successful tests in a phase should reinforce the improvement of the solution for the next phase of testing. Failure of tests in a phase should encourage a deeper study of problem formulation and possible adjustment to the solution and test scenario. The outcome of this stage is that team has identified the solution that should be used for regular value delivery. In addition to the skills in the exploring stage, a team would need at least a couple of experienced data scientists and a data engineer to be functional. The team also needs some knowledge and experience in hypothesis testing and experimenting. At this stage, it is important that the technical solvers, work in close collaboration and stay broad.
  • Industrializing: In this stage, the team should focus on improving the main solution in terms of achieving a decent level of efficiency and automation. The goal is to attain sufficient technical depth to ensure regular delivery to a small to a medium-sized target area. At this stage, the solution may encounter edge cases that emerged while being in operation. Therefore, the assumption and the design of the solution should pivot here and there. Therefore, the technical sophistication of the solution should not come in the way of the ability to experiment with the solution. Apart from the skills in the previous two stages, the team needs to complement the data engineer with software and machine learning engineers. It is important to maintain at least two engineers for every data scientist or analyst in the team. The team can also afford to have a few inexperienced team members. It’s important that the team takes operation responsibility seriously, i.e., be well prepared for upcoming runs and willing to drop anything hand to get runs unblocked. Furthermore, it is important that the tasks are delegated to individual contributors with strong encouragement for occasional joint work. To ensure that someone needs to step up as a tech leader who would facilitate the discussions architectural and algorithmic design and implementation as well as team culture. At this stage, the technical roles need to stay broad, then digging deeper.
  • Scaling: In this stage, it’s all about enhancing the capability of the system to a much larger scope to support the majority share of the business. Part of solution design may need to be reimagined to work with the scale. Furthermore, the technical sophistication of the solution needs to be very high in terms of automation, efficiency, and transparency. It is also important that the development, testing, and production environments should be similar for ease of transferring solutions. Once a team matures in this stage, the development activities should reduce a lot compared to operational support. Apart from the skills needed in the earlier stages, the team needs to increase the headcount in all roles. In addition, the team should focus on getting a few specialists in the area of big data, cloud, mathematical modeling, software development, etc., at least part-time.

Ideally, a team would go through these stages linearly. However, we do not live in an ideal world and, therefore, such a narrative does not really fit. There will be changes in the business settings, which will lead to continuous pivoting here and there in the solution. In reality, as a team matures, it will have different workstreams in different stages.

Ten practices to form the right team

It is very difficult to recruit people with specific qualifications all the time. Teams and organizations compete for the same talent pool, which cannibalizes the market. Furthermore, the lower the maturity of the team, the much worse they seem in recruiting. Finally, the recruiting budget is always behind the expectation of the candidate. When the team is recruited, start considering ML system delivery similarly to deliver a soap opera for decades. You need a lot of people doing more than their jobs. You need people rising through the ranks. You need new and fresh faces on a regular basis. You need multiple people to take a lot of ownership and coordinate the process. Skills and experience aside, getting the right mindset is a must.

While it is tempting to share whats should be the way, let me share a few tips that worked for me and my teams to get into the right mindset.

  1. Recruit a few ninja generalists early: The first few hires need to do everything by themselves. It is necessary that those few would have the capability to work with the full stack collectively. These few people will build the nucleus of the team and, hence, need to be highly collaborative.
  2. Recruit specialists who are into mentoring and/or good culture: It is bad to hire specialists who hide in the corners and do their own thing. The person should inspire and help others to be better in their skill areas. It is important that these people drive major impactful causes, e.g., reducing technical debt, increasing transparency, being experimental, etc.
  3. Take a chance: Often hiring requirements does not match exactly with a potential hire. In that case, take chance as longs as there is enough attitude and desire to make up for the gap. For example, if the position is for a Python software developer, but the candidate has Java experience and is willing to learn other languages, go for it.
  4. Target diversity: Avoid filling your team with super senior like-minded people. Machine learning is all about optimizing, which comes typically when global or wide-scale is considered. In that landscape, it does not matter how sophisticated the solution is if it is not robust. What matters is its capability to cover niches in addition to the mass demands. When a team stays homogenous far too long, it is likely that it will struggle to pivot, whereas a diverse team is likely to move in the right direction due to its variety in thinking. Try to see recruitment as a puzzle that needs blocks of different colors representing experience levels, education, gender, ethnicity, etc.
  5. Encourage agility: Achieving agility requires free time to think, learn, and react. It is a must that the team leaves room for upskilling and surprises while delivering. Plan no more than seventy percent of the team’s capacity and encourage teams to spend more time pair programming, studying, and discussing. Encourage juniors to take more ambitious work, seniors to lead, people to diverge more than their typical responsibility. Ultimately this will lead to individuals taking more ownership of their work and, hence, achieve the mindset and ability to react faster.
  6. Reduce tribal knowledge: Always keep asking can we do what we do while being fault-tolerant. If not, reduce high dependency on individuals no matter the complexity of the work. As soon as the team reaches a steady ground of its solution, rotate to increase knowledge, give responsibility to a group than to an individual, make documentation and demo mandatory.
  7. Pursue a good ecosystem: Avoid just running a team delivering a result. Take sufficient detours, such as participating in hackathons within and outside of the team, join forums, support interns, run side projects, and rescue other teams. We cannot buy our way out of crisis all the time. For that, we need to be social where the community solves the communities’ problem. Think of the challenge from the point of view of an ecosystem than your team. If you have benefitted from another team consciously or subconsciously, you should return the favor consciously.
  8. Build a safe space for sharing: It is common for us to share and reward achievements. Such things inspire us to do better. However, that is not good enough. It is also important to reward and encourage good habits through positive feedback. For example, if someone has written good documentation or covered code with a good amount of tests, praise the person publicly during the code review. Furthermore, the team should highlight learning from mistakes and really try to find the root cause without blaming an individual. For example, if someone made mistake because of a lack of knowledge, it’s a failure of the team that they have not considered the education of the team member. Making mistakes is common and should not be punished. What matters is what the team does once it identifies the mistake. Sharing the root cause and suggesting a remedial process is not only beneficial for the team, but also for other teams.
  9. Strive a learning organization: Encourage active learning and sharing. Allow sufficient room, budget, and means for people to pursue all sorts of learning activities, such as taking classes, explore new technologies and methods, form study groups, form learning councils, encourage certifications and degrees. Competence growth has a snowball effect, which comes from a friendly environment and a critical mass. It is necessary to work towards that from the early stages even though it has no immediate return.
  10. Create an easy atmosphere: Most importantly, adopt doing good, be kind, and reduce misbehavior. Create a steady where most of the time it will be stressless. Induce events that make it is easy to do so. In Swedish culture, where I work, it is encouraged to have Fika, a daily event, where people have coffee, tea, bun, or banana and chitchat about arbitrary things several times a day. I strongly recommend that to everyone.

Remarks

I have tried to work on each and every practice in some capacity and observed improvements. I am not a researcher in this area and have no numbers to back my arguments. If you have stronger evidence that supports or goes against my argument, please share. I will appreciate the knowledge. If you have other suggestions, please share them too.


Related Articles