The world’s leading publication for data science, AI, and ML professionals.

Spotting Talented Machine Learning Engineers

An understanding of key skill areas to identify talented machine learning engineers. Such an understanding will help in recruiting…

Figure 1: Disciplines relevant to a machine learning engineer. Figure by the author.
Figure 1: Disciplines relevant to a machine learning engineer. Figure by the author.

Machine Learning Engineer (MLE) is one of the hottest roles these days. While many would associate such a role with Python, R, random forest, convolutional neural network, PyTorch, scikit-learn, bias-variance tradeoff, etc., a lot more things come in the path of these engineers. Things that an MLE needs to handle does not only derived from the field of Machine Learning (ML) but also from other technical and soft disciplines. As depicted in Figure 1, in addition to possessing ML skills, an MLE needs to know programming, (big) data management, cloud solutions, and system engineering. Furthermore, the person needs to have quite a lot of project management skills as well as be a solid team player without sacrificing personal curiosity and ambition.

In this article, we share how to spot talented MLEs in terms of capabilities on those disciplines. We divide the disciplines into four categories, where only one of them is purely technical. The rests are more soft, organizational, and project management skills, which may be relevant to other professionals. We also share the evolution of the expectations as the maturity of engineers evolves. You should consider that the engineer at a certain level of maturity acquires new skills in addition to the skills achieved in earlier stages. In the article, we only show three levels: junior, senior, and lead. If you do not agree with the levels and would prefer more granular levels, adapt as you see fit.

Scalable Machine Learning Systems

As illustrated in Figure 2, supporting an ML-based system needs a lot more than the ML code. Therefore, MLEs need to know concepts from Data Science, computer science, software engineering, cloud engineering, and systems engineering. Furthermore, MLEs need to be fluent in programming languages, libraries for data science, libraries for big data engineering, cloud and system management platforms. In this setup, MLEs perceive machine learning code beyond notebooks or scripts. Rather they see them as applications that can be modularized to make it easy to work within production infrastructures. They also know that ML codes need to be hosted, managed, executed, debugged, logged, and monitored using a set of systems working together. We capture the capabilities of MLEs in these dimensions.

Figure 2: Only a small fraction of real-world machine learning systems are composed of the machine learning codes, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex. An MLE needs to delve into all these systems. Originally published in the paper Hidden Technical Debt in Machine Learning Systems.
Figure 2: Only a small fraction of real-world machine learning systems are composed of the machine learning codes, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex. An MLE needs to delve into all these systems. Originally published in the paper Hidden Technical Debt in Machine Learning Systems.

Fundamental Knowledge in core technical disciplines

  • Junior-level engineers possess functional knowledge of these disciplines.
  • Senior-level engineers possess wide and in-depth knowledge in some of the areas. These engineers can also mentor juniors to improve their knowledge.
  • Lead-level engineers can redefine concepts and knowledge in some of the areas to suit more to the problem domain. These engineers can also mentor seniors on how to become more specialists or generalists in those disciplines.

Knowledge of libraries and frameworks in core disciplines

  • Junior-level engineers are fluent in well-known libraries and frameworks.
  • Senior-level engineers have worked a lot with many alternatives and understand the design principles and underlying implementation of some of them. These engineers can also help teams in adopting and demystifying the libraries.
  • Lead-level engineers have bootstrapped or contributed to some well-known ones used by many. The engineers can also identify libraries and frameworks that should be on the radar for the teams.

Implementation of machine learning workflows end-to-end

  • Junior-level engineers know how to implement typical end-to-end ML flows.
  • Senior-level engineers know how to improve/redesign/reimplement components in the workflow for large-scale or complex deployments. The engineers also know how to test and validate the flows.
  • Lead-level engineers can come up with reference architectures, their proof of concepts, and enable other teams to adopt the architectures.

System design and implementation

  • Junior-level engineers know how to interact with operational and development systems.
  • Senior-level engineers know how to implement some of these systems using common virtualization, DevOps, and cloud technologies. They can also mentor the junior engineers to become more proficient in using and mastering the building blocks.
  • Lead-level engineers can design reference architectures and their PoCs. They can also help the teams to strategize the adoption of those architectures. They also know implementation approaches for fault tolerance, testability, debuggability, etc.

Best Practices

Like any other engineering, machine learning engineering is a team activity. That basically means certain practices need to be fine-tuned so that teams can interact with each other effectively to make it easier to deliver. MLE also needs to be able to prototype a lot. Furthermore, MLEs need to develop things that are easy to share and have to be willing to build on someone else work, which comes from an inner source culture (imagine adopting the open-source culture, only within the bounds of an organization).

Engineering best practices

  • Junior-level engineers are good at carrying out the tasks that adhere to the principals
  • Senior-level engineers help teams to adopt certain best practices while continuing value delivery.
  • Lead-level engineers can flesh out the principles into actionable practices. They can also figure out the (long-term) strategy for the teams to climb up the ladder of technical maturity through the adoption of key best practices without sacrificing current value delivery too much.

Prototyping

  • Junior-level engineers can occasionally start a new component by looking at known examples.
  • Senior-level engineers can implement/validate new components without having a known example.
  • Lead-level engineers can come up with design principles and processes to implement/validate new components and drive the development of core components to enable rapid prototyping.

Inner source

  • Junior-level engineers know how to contribute to the inner source.
  • Senior-level engineers can drive some of the key best practices and tribal knowledge documents. The engineers can help teams to translate those in different contexts.
  • Lead-level engineers can identify a framework of inner source culture that would enable senior engineers to draft the content of the best practice and tribal knowledge documents.

Project management

Like any engineers, MLEs need to be efficient at their day-to-day work. This means mastering project management skills including design thinking, planning, understanding delivery, and agile principles. This is particularly true for MLE since it is likely that an MLE is placed in a central role within a cross-functional team and needs to pull in a lot of weight.

Design Thinking

  • Junior-level engineers know how to perform simple requirement analysis.
  • Senior-level engineers know how to perform thorough requirement analysis, validate requirements, and helps teams to flesh out the details.
  • Lead-level engineers build strong relationships with stakeholders. They know how to engage in a dialogue that maps the journeys of the stakeholders and translates them into a set of system requirements. The engineer also knows how to define metrics in order to understand the requirements and their solution analytically.

Communication

  • Junior-level engineers communicate well with the team whenever there is a need.
  • Senior-level engineers communicate well with the stakeholders as well whenever there is a need.
  • Lead-level engineers strive to establish environments for easy communication.

Planning

  • Junior-level engineers know how to implement a plan for an engineering problem.
  • Senior-level engineers know how to flesh out the details of such plans and refine the implementation details.
  • Lead-level engineers know how to define milestones for engineering problems and help teams to prioritize activities to ensure that the milestones are met. They know how to make the right trade-offs and do not over-engineer things.

Delivery mindset

  • Junior-level engineers know how to deliver as planned within a reasonable time.
  • Senior-level engineers know how to break down deliveries into manageable sets and can support stretch goals, e.g., reducing cost, higher efficiency, etc., for focused work.
  • Lead-level engineers know how to drive concurrent delivery agendas with complex scheduling. They also anticipate upcoming situations and progress on the runway sufficiently to make it easy for delivery.

Agile principals

  • Junior-level engineers typically become fluent at ceremonies of an agile method (Kanban, Scrum, etc.).
  • Senior-level engineers know how to adhere to the principles and help teams to stick with them.
  • Lead-level engineers know how to translate the principles of agile ceremonies for teams and help them to choose the right variants

Culture fit

MLEs exist to support teams, not the other way around. Therefore, a lot of emphases should go on how MLEs would fit in the team’s culture. It does not mean that MLEs should agree on everything that the rest of the team decides. Rather MLEs, like any other professionals, should do things that make the team better, even if it means disagreeing with the rest of the team, but complying with work packages anyway. Furthermore, whatever the engineers are driving, they should take full ownership as much as possible. Striving for diversity is the way to achieve a healthy team that knows how to deal with complimentary opinions. Machine Learning is meant to solve complicated global problems, which is not possible to pull off with a narrow background team. Most importantly, engineers need to have a growth mindset, which obviously comes from active and passive learning.

Team first

  • Junior-level engineers are ready to step in whenever the team needs them.
  • Senior-level engineers know how to step in and can adjust the level of engagement as the situation changes.
  • Lead-level engineers actively work towards creating an environment for nurturing healthy teams.

Ownership

  • Junior-level engineers know how to take ownership of areas that they feel comfortable with.
  • Senior-level engineers know how to take ownership of all sorts of areas when they are asked even if they do not feel comfortable with some of those.
  • Lead-level engineers know how to take ownership of anything no matter the level of comfort and establishes an environment that encourages others to do so.

Inclusiveness

  • Junior-level engineers know how to work in a diverse working place.
  • Senior-level engineers know how to support the team to stay inclusive.
  • Lead-level engineers know how to work towards a more inclusive working environment.

Learning mindset

  • Junior-level engineers have a lot of drive to upskill themselves.
  • Senior-level engineers know how to foster learning activities for a group of people while satisfying their own learning goals.
  • Lead-level engineers know how to enable learning for groups through processes and strategies.

Remarks

I am sure I have not fleshed out many other important attributes. However, at least the ones mentioned above would get you started. If you do not agree with some of the above points, please share your views. If you feel that some other points need to be mentioned, please share your views. If you agree on the points and found them helpful, please share your views.


Related Articles