How to Price an AI Project
I have been asked many times by clients to provide fixed price estimates for large Machine Learning (ML) projects. This is really tricky. Requirements often change midway through a project as a result of feature creep, development slippage, integration headaches, user acceptance, and many other factors. I advise clients not to fight requirements changes in their first ML project, which is the exact opposite from traditional software development principles. Machine learning is not regular programming. It is basically applied data science, and rolls out very differently in an organization that already has a non-machine learning infrastructure as compared to a start-up with a clean slate.
The bigger the unknown, or what I like to call requirements risk, the more I lean toward hourly rather than fixed price. Where requirements are very tight, or what I like to call an executable specification, the more I lean towards fixed pricing.
In a previous article I discussed how to hire an AI consultant, and in this article we will look into how a project gets priced.
A government rep we work with likes to call these candid conversations the open kimono approach. Brutal honesty. Let’s be realistic about the cost of entry and hourly rates. Our standard rate in August 2017 is $250/hr (USD) plus travel and per diem if it is an onsite gig. Given the effort to start a contract, including an NDA, statement of work, and initial discussions, the contractor is aiming for projects that are at least 5K (20 hours) in size. A one hour project with low potential for followup is just a non-starter (e.g. debug my code). You (the client) also have budgetary concerns and need to account for internal resources. We like to delegate as much lower-end work as possible to the client’s staff to lower the hours required to complete the project. But at the end of the day, you are sill paying your employees to work on this project, and need to take that into account. Your goal is to cap the cost and time for your ML project. Our goal is to try and hit our hourly rate, even on fixed price projects, and to gravitate towards big projects wherever possible.
The purpose of this article is to help you price out your project at a high level. Let’s price a small made-up project and see where we end up.
CLIENT REQUIREMENTS
Critical to any project in ML, we need to set expectations about accuracy, precision, recall, F-score, etc. Let’s assume the client wants “good” results rather than a quantitative benchmark. That makes the project easier to price out, as there are fewer iterations. In general, the quality of the AI grows with the size of the dataset.
Another dimension to consider is requirements quality. Many clients show up with poorly defined requirements, which is why they need our expert advice not only for the work, but also for the project definition and solution architecture. It is critical in the earliest possible stage of discussion to identify any provably impossible to achieve requirements, or requirements that require new science. No kidding. I have had requests to fix-price research projects, when the definition of research is that you don’t know the answer. Our solution is simple. To fix price research, we provide blocks of hours at a fixed price. Results are not guaranteed, but at least the burn rate is controlled. In those projects we aim to deliver results as early as possible, to motivate further development.
The imaginary project we will price in this article is a document sorting system that gives BI analysts in a financial institution the ability to dump documents (PDF, Word, etc.) and sort them by semantic and syntactic similarity in many different ways. The client is interested in similarity between documents, and also similarity between each document and the keyword contents of a search box, and also similarity between the documents and a user-pasted paragraph of text. This is a common generic task for analysts, and not a super novel ML project, but it gives us something we can price easily. Let us define that the client has no ML server. The data to be sorted is not sensitive internal data, and so it can be shipped securely offsite into AWS for the ML system to digest and present. As I discussed in a previous article, some companies have trouble using public cloud on their data for various reasons.
Let’s plan this thing out, and then estimate a price.
PLANNING
Step 1: Lay the foundations
We now know the project will live in AWS, and will use one or more p2/p8 instances spun up from an existing AMI. Assuming, in conversation with the client, the custom elasticsearch features won’t cut it for these analysts, and the regular search features are also necessary but not sufficient, we decide to build a hybrid model of ML and existing document search tools. The system will need tensorflow, word2vec, doc2vec, keras, elasticsearch, and Flask to expose this set of features as a RESTful JSON service like elasticsearch does. We will need a PHP server for an AngularJS GUI (maybe even on the same instance if it is a small-scale project). We decide against NLTK, because elasticsearch covers all the stuff we would use from that package. Jupyter notebook is an excellent idea, so that the developers who will own this project can easily and securely run code on the servers, but also so that we can reuse code from github examples with lower effort. It’s a win-win. We also need to know that the client requirements map to tools like this:
Document similarity + document to paragraph similarity = word2vec and doc2vec, also some Keras models for CNN+word2vec. FYI this is just one of many many examples we can start with and shape to our needs.
Document keyword search and other search features = elasticsearch
Reusing models saves time and money. We will likely use the pre-trained google news word2vec model, or maybe a gloVe model. Basically the plan is to use some code we have on the shelf.
Word2vec does not do transfer learning on saved models, and so pre-trained models are perhaps not a good representation of how well a custom model will perform. We advise the client, and they agree with the plan.
elasticsearch has many cool APIs that the client does not know about, and so we schedule some training for the client’s staff in addition to the dev work.
That’s the high level.
Step 2: Prioritize early results
So we want to price this project. Let’s decide what the most urgent priority is for the company, and price the first phase of development. Why decide on the terms of our marriage before we have had a first date? My philosophy here is to give realistic pricing from day 1, and keep the initial phase short so that the client can see and feel the results quickly. The project is about semantic similarity of sentences, paragraphs, and whole documents. Let’s get the system up, and show the client the features they can’t get off the shelf. The client agrees that we will put up 2 instances. One server we call “WEB1” is the RESTful LEMP web server that will contain our Angular code, the file dump, and later it will also host the elasticsearch code. No GPU on this instance, but it will have a big block storage attachment to hold all the files. The other server we call “ML1” which will hold the ML system. The ML server will accept jobs from the web server and will store responses that will be dumped in batches back to the web server. ML1 will be a p2 instance with high memory as it will be hammered with lots of GPU jobs, but also with high memory demands. This is because word embedding models like those we will have in word2vec/gloVe are more memory intensive than compute intensive.
We did not capture topic modelling in this first phase, and also threw overboard for now the whole elasticsearch module and training. In conversation with the client, we also push document relevance regression. These items can always come in for phase 2. Right now the client wants to see the magic happen.
Next we throw the project tickets into JIRA.
Focusing in on document classification, we estimate 10 hours for AWS instance provisioning, and another 5 hours for certificate installs (PKKs, SSL, OTP, etc.). This 15 hours includes the tool installs. We add 3 hours of knowledge transfer to the quote to make sure the client knows how the system works. Now we have a working 2 instance system. Let’s set up the ML and GUI. No need for load balancers, geographic distribution, etc. We keep it simple.
We estimate 10 hours for the GUI development including the file dump and presentation of the results. We decide to use tensorboard to show the document clustering, and an angular table to rank results in a separate GUI. Authentication is by HTACCESS password for this phase. In phase 2, we can do Oauth 2.0 login so that the analyst can log into the system with their work account.
So now we are at 25 hours. Let’s keep going.
The core ML will not take more than 10 hours to configure and deploy. The requirement are very tightly matched with the existing capabilities of ML tools. This time includes an hour demo for the client to show them how the whole system works.
Step 3: Account for slippage
The rule of thumb is to estimate a factor of π when trying to approximate a project price or duration. Let’s truncate pi to 3. So slippage can be up to 35*3*(2/3) = 35*2 = 70 hours.
PRICING
Server setup: 15 hours
Interface setup: 10 hours
ML development: 10 hours
Slippage: up to 70 hours
So, we estimate a phase 1 effort of 35 to 105 hours, or $8,750 to $26,250. To keep costs contained, I would walk through the steps the client can take to avoid slippage, including planning the network access ahead of time (firewalls, ports, certificates, etc.). This is the sort of project that I would offer to fix price at 10K, because the risk of slippage is low.
If we specify a second and third phases, I expect they would each come to about 10K, for a total project cost of 30K. The phases would be released approximately each 4 weeks. Maybe faster as this project is significantly more straightforward than average. It is only at this point where the project is specified as a statement of work that the client actually signs the contract.
As we close out the project, we try to upsell more ML solutions to the client, which can add value for the client beyond the initial project. This keeps us in the game for phase 4 and beyond.
CONCLUSION
Hopefully this has been a really deep dive for you into the mind of an AI consultant when it comes to pricing projects. If you like this article then please recommend it, share it, or give it some love (❤).
Happy Coding!
-Daniel
daniel@lemay.ai ← Say hi.
Lemay.ai
1(855)LEMAY-AI
Other articles you may enjoy:
