The world’s leading publication for data science, AI, and ML professionals.

How to Hire an AI Consultant

So, you know you need some machine learning/AI development. Why not just do it yourself? Well, the short answer is that specialists have…

So, you know you need some machine learning/AI development. Why not just do it yourself? Well, the short answer is that specialists have more experience than generalists, and can therefore reduce your risk and effort. The longer answer is about knowing where the land mines are hidden, and what good requirements look like. I’m a machine learning expert, and I perform contract work as a consultant to companies in the U.S. and Canada. I also have significant experience developing businesses. Integrating the lessons I’ve picked up by working in the field, let’s talk about hiring an AI consultant.

If your project is not funded, your best bet is to try and work with a university lab to get some early results. Let’s assume for this article that your project is a standard case of a business adopting AI to improve the core product offering. This is therefore a "small R; big D" R&D project. Some projects, especially startups, are a "big R; big D" affair, and that is a topic for another time. For now, let’s stick to the assumption that you have a team in place and customers to server, and want to add some cutting edge AI capabilities.

You are looking for an AI consultant. Let’s now get more specific about what to look for. You want a consultant who is more than a data scientist. You need engineering. Machine learning projects rarely involve just the data science part. You need to aggregate data and massage it into the right format for the AI to learn. Sometimes the project involves hardware, wireless, and networking. This is the cloud and the Internet of Things (IoT). It also helps to hire a consultant that understands the business implications of your requirements, kind of like a management consultant. You want a resource that can talk about budgeting, planning, and solution architecture, knowing what a machine learning deployment at scale looks like. It is also a good idea to involve grants wherever you can, as the ML field is very cutting edge, and should qualify your R&D for some government funding programs (In Canada, these are Mitacs, OCE, NSERC, etc. etc.).

Another aspect of hiring a consultant to get your AI program off the ground is capacity building. You want to build into your company some generic infrastructure to provide the capability to do future machine learning projects, and to instill the knowledge within your devops team(s) to maintain and extend the code brought in by the consultant (as much as possible/practical). Building an AI capability into a company can be a program management and team building process as much as it is a software/hardware project. It often involves board meetings, linking these goals to Key Performance Indicators (KPIs) and stakeholder meetings. It is a smart idea to have the exec(s), product manager(s) and senior developer(s) in the same room when the project is being laid out. I have prepared many proposals, interacted with investors and key opinion leaders, and presented product to large audiences. No two meetings are the same. And that’s why it’s good to have a specialist on your side when you decide to get your company into AI.

Ops and Big Data

"Fortune favours the prepared mind" (Louis Pasteur).

Ask any prospective consultant about their recent job history. Even with NDAs in place, there is some information they can share that is not embargoed. For example, "Describe a recent project you did for a large customer" can be answered with "I recently completed a project involving the architecture and deployment of a big data AI solution for a US client. It was an on premise solution starting with the barebones infrastructure server running Ubuntu server with GPUs, and ending up with a service capable of supporting many thousands of transactions per second. They were happy with the results. I’ll ask them if I can share more details with you." This is the sort of response to expect. In my case, I have publicly verifiable patents, articles, and a doctorate listed so that clients can understand the level of my work. My recent work on machine learning will be presented in July 2017 in Nice, France ("Unsupervised Deep Learning Recommender System for Personal Computer Users" and "Visual Deep Learning Recommender System for Personal Computer Users").

In a followup you might get something like the following from a good consultant. "Moving back towards the tie in between big data and cloud, I want to address my recent experience with cloud connectivity of data streaming physical assets. My team helped a recent client to connect hardware units to a cloud system that logs, analyzes, and presents data visualization. We designed the cloud solution from the ground up, and expect in a subsequent statement of work from the client to involve our adding machine learning analytics to their live production solution." This tells you that they know cloud and IoT, and are comfortable getting into AI at scale.

Most of my work is under very strict NDA, and so I have provided examples here from my academic work or work that is in the public domain to illustrate how I work in practice with clients. But… Don’t let a consultant hide behind NDAs. Especially if they can’t produce for you statements like those above, your consultant should have a public profile that shows where they are coming from. For example, peer-reviewed publications, relevant university degrees, patents, panel discussions, a team, and so on. Using myself as an example, I have completed a masters degree on the subject of finding which parts of a software program should be accelerated in hardware given a hardware constraint. My PhD was on recommending actions to users with unsupervised learning. This all comes together to tell the client a credibility narrative. Look at the consult’s website. Is it nice, or are they just an email address and a blank page? It goes without saying that the consultant should be incorporated. Moreover, their company website is a nice way to judge how organized the consultant really is. Do they basically do anything for money (e.g. will code for cookies), or are they narrowly focused? For example, in our case, we do IoT and AI, and that’s it.

Expect to encounter a process. Consultants love process. With requirements gathering and whiteboard sessions expect to dive deep into how the solution would be structured before writing any code. You should discuss how the system would be used, and the business case for how would it would be sold to the customer. The consultant should build a plan with you called a Statement Of Work (SOW), with milestones and deliverables, and then build stakeholder buy in.

Each consultant deals with risk differently. My approach to solution architecture is organization. Early on in my career, I was working directly in operations, and datacenters and network operations are very serious places that continue to give me pause. If one person pulls the wrong chord in ops, or deploys one bug into those servers, then every customer could be calling support. It is a high pressure environment. In response to that pressure, I follow the traditional system architecture approach to dev teams, starting with dev, integration, and production, and including rollback options, deployment verification with QA, and regular VPS backups at each distribution centre for risk mitigation. Wherever possible I recommend clients use the cloud to avoid this sort of ops risk.

I am a big fan of cloud solutions. Machine learning code runs on GPUs that are sometimes cheaper to buy and run than to run on AWS. The key value I bring to the table is the ability to comprehend and design the software architecture as well as the hardware required. The advice you want, is when to go to the cloud. Not all clients listen, but all consultants should be able to help you decide where to build your system (cloud or on prem). I routinely provision GPU hardware into client infrastructure, be it cloud based in AWS, in a dedicated rack in the data center, or on premise in the client’s private facility. GPU hardware setup and deployment into production environment is something I understand. Specific technologies that are good to mention here are Nginx for serving and load balancing, memoization for caching results to reduce both latency and unnecessary compute, replication and indexing to take the pressure off of high-traffic database tables, and also distributed query systems such as noSQL. My personal preference is for postgres, although I have worked with t-SQL (SQL server), mySQL, and Sqlite. Experience has taught me to normalize tables, and listen to the DBA, because he is smarter than everyone else when it comes to databases, and to squeeze performance from data outside of the database with preprocessing wherever possible. Any good consultant will make limiting statements like this, declaring that they are not an expert on everything. If a consultant always claims expertise, regardless of the topic, then you should worry.

Any AI consultant you hire must understand security. They are often touching your most sensitive data. Through my engaging in several big data projects, I have developed an intuitive feel for cloud automation that jives with security needs. I view the cloud, be it private or public, as a VPS management system. Many base technologies from authentication (e.g. Oauth2) and authorization (e.g. role based access control pattern) to SSL and firewalls are required in order to stand up a production system for a company. I do this frequently, from scratch, and I am familiar with security systems involving hardware and software (See Approved Patents) and the threat models and security policies that go along with safely developing a project like the one described in the posting for this position. For example, whitelisting and blacklisting IPs, PPK authentication, RSA token-based authentication, etc. Of course any system is only as strong as the people who use it. Training staff to comply with security policies is essential to maintaining the security infrastructure.

Machine Learning

Let us now dig into the key topic in this discussion. Namely, can the consultant design and implement a machine learning system to meet your needs? How can you tell?

Visualization of the output from an image processing neural network classifier in 3D space.
Visualization of the output from an image processing neural network classifier in 3D space.

As I described in a recent post, machine learning expertise is hard to find. General knowledge is quickly reaching developers, which is excellent, but deeper expert knowledge is harder to get from a short online course, or online video. Don’t get me wrong, I love those resources and highly recommend them, but they are not enough to give your dev team the skills they need. Looking back at my files, I took my first Artificial Intelligence university course in the final year of my undergraduate degree in 2006, before it was all the rage. In parallel with the effort required to complete a PhD on machine learning on a part-time basis from 2009 to 2017, I have been working a series of very interesting jobs related to machine learning, going all the way back to my work at Zip.ca (Momentous Corp) where I designed a recommender systems and conducted operations research (e.g. integer linear programming, combinatorial optimizations, knapsack problem, transhipment problem, and others). My task at Zip,ca was to find a "good" assignment of DVDs to customers, given a list of customer wants and DVD availability at each distribution centre. This work was competing head to head with netflix in Canada.

With a consultant you don’t know, always look to start with a small proof of concept deliverable to prove to yourself that this consultant knows their stuff. Work with the consultant to come up with a project that is a low hanging fruit. Something that they can deliver on quickly without much development effort (e.g. based on existing code they already have, and data you have already collected). If this first step goes well, then you can confidently move to a bigger project scope.

Don’t think too much about what tools the consultant is using, as long as they are industry standard. I do most of my Machine Learning work in python, and my frameworks of choice are Tensorflow 1.0, and scikit-learn. I sometimes use Keras. Almost always my code runs in a GPU. Wherever possible I reuse existing machine learning models rather than building them from scratch. Expect the consultant to show you how to save everyone time and money by reusing existing AI code.

Ask the consultant to describe a deployment, in general terms. The example I would give for an image processing project goes something like this: "I have employed Convolutional Neural Networks (CNNs) and supervised learning for discriminating between classes of images based upon labelled data. For example, discriminating between wounds on a patient, detecting differences between produce based upon appearance, and other such image classification tasks. In production, I would use a GPU-capable VPS and a server such as Flask and a load balancer (nginx) to make these models high-throughput. Training a full CNN can be time intensive, and so as a first step to get to know the observed features, I have used transfer learning from an existing CNN to see how well the features of interest in the labelled data cluster into the desired categories. Outside of supervised learning, I have deployed unsupervised learning models when there is insufficient labelled data for supervised learning. Sometimes dimensionality reduction is appropriate for unsupervised learning (e.g. for clustering I use t-SNE and sometimes PCA or K-means)."

Whereas, for a text processing project, I would respond with something more like this: "I am a very big fan of word embedding models. For this type of task, I use word2vec and/or GloVe, either with Tensorflow or directly in python, and leverage existing models such as google news vectors and wikipedia, or build my own model based upon a text corpa or the client’s dataset. There are many other machine learning models that we can go into in greater detail, including generative models such as RNN and LSTM, and newer learning models such as sentiment analysis, and more."

Conclusion

Hopefully this article will help you decide who to work with as a machine learning consultant. The take home message is that you hire an AI consultant to be more than a data scientist. You want them to provide assistance in designing an overall system from requirements to deployment, and help with capacity building so that your internal resources in the devops team(s) can support the AI infrastructure and code. Expect the consultant to have some process for working with you, including a Statement of Work that sets clear expectations. Ask a prospective consultant tough questions, and make sure you get some sort of recent job history and high-level validation that they understand the risks, security, and cloud components of your project. Ask them to describe a deployment in general terms, and try to get them to reuse existing AI code wherever possible, saving you both money and time. Look into government grants and how they apply to your project. Finally, even if the consultant seems awesome, start with a small proof of concept, to make sure they can deliver on your expectations.

Happy Coding!

-Daniel [email protected] ← Say hi. Lemay.ai 1(855)LEMAY-AI

Other articles you may enjoy:


Related Articles