The world’s leading publication for data science, AI, and ML professionals.

Should you use off the shelf or develop a bespoke machine learning model?

An argument for and against using prebuilt off the shelf models.

As a data scientist, I love exploring new areas to apply my skill set. Figuring out what model will fit a problem area and thinking of some features that could describe the desired outcome are challenges I enjoy. With the rise of off the shelf (OTS) models, there is now more automation in this space and this raises the question of whether you should bother building a bespoke model to solve a problem. This doesn’t mean that OTS models are taking over and there is no space for bespoke models, they are both tools and both have situations where they’re required. As a clarification, I will not be touching on AutoML or MLops platforms which although they’re off the shelf systems they’re not specifically models, and are both topics for later articles. In this article, I am going to explore the pros and cons of both OTS and bespoke machine learning (ML) models and discuss when you should think about using OTS and when you should go ahead with developing something bespoke.

Firstly, let’s define what an OTS model is: OTS commonly refers to a product that is available immediately and does not need to be specially made to suit a particular purpose [https://dictionary.cambridge.org/dictionary/english/off-the-shelf]. Within ML and AI this usually means a model or API (with a model behind it) that is prebuilt, pretrained, and preoptimised for a particular task. The data inputs for these models are defined, you have to conform to the provided schema, and the data outputs are usually fixed due to the particular task the OTS model is built to solve. Which doesn’t always align exactly with the problem you are trying to solve.

Examples of OTS models in the cloud are Azure Cognitive services [https://azure.microsoft.com/en-gb/services/cognitive-services], and Amazon Rekognition, Amazon Comprehend, and Amazon SageMaker JumpStart (to name a few of AWS the offerings, they don’t have a collective name) [https://aws.amazon.com/sagemaker/features].


Advantages of using off the shelf.

To start with the positives, OTS can be low cost in comparison to bespoke models. This is mainly because engineering time is costly, using an OTS model can seriously cut your development time and therefore reduce development costs. There are still some costs in the form of subscription or processing fees charged by service providers, which are pay for what you use.

The other main advantage is accuracy, though this is heavily dependent on your use case. If an OTS model fits your use case exactly and has been pretrained and preoptimised then you will struggle to create a bespoke model that competes. This is mostly due to service provides often having large teams working on these models and having access to significantly larger amounts of data from all users of the service. So, if your use case fits the use case of the service, then it can be a good idea to take advantage of that.

OTS models are really useful for greenfield or proof of concept (POC) projects (a project that lacks constraints imposed by prior work). They allow you to get started quickly and produce results in a very short space of time, especially if you have a lack of your own data.

When a system goes into production, maintaining that system can be a challenge. More so if you have a bespoke model within the system that needs productionising. Using an OTS solution will allow you to move into production and make use of the production-ready nature of many OTS services, limiting your need to maintain the model so you only have to worry about the connection to the service provider’s API.

Disadvantages of using off the shelf.

The problem often arises that your use case is slightly different or not covered by the OTS service. If your use case is slightly different, it might make sense for a POC project to tweak the requirements, or perform some post- or pre-processing to the data to make it fit. This will impact your accuracy and could mean you consider a bespoke model in the future. In the case that an OTS service doesn’t cover your use case, then you are stuck without a choice but to create a bespoke model. As most ML systems consist of multiple stages and models, it may be likely that you can use OTS services for part of your pipelines. It is worth the exercise to break your system down and see if an OTS model fits the needs of one stage, especially in the early days of a project.


Advantages of using bespoke.

Accuracy is often touted as the reason for using a bespoke model, but as discussed above OTS models can outperform a bespoke model if your use case fits. Often OTS doesn’t fit well and has to be shoe-horned, this can be ok for a proof of concept, but a bespoke model will quickly outperform in this scenario.

With bespoke models, you are able to be more flexible and apply any cutting-edge techniques that might become available a lot sooner than waiting to see if your service provider will implement any new techniques. With this flexibility also comes the ability to experiment and try different ways of solving your specific problem.

With a bespoke model, you get more control over the privacy of your model, as the data is not being passed through your service providers system and therefore you don’t need to be concerned about their stance on privacy.

Disadvantages of using bespoke.

Building truly bespoke Machine Learning models can be hard, very hard. Especially when it comes to feature engineering, optimisation, and productionisation, there is a lot to consider. Most machine learning systems require a subject matter expert and a machine learning expert, and as with all inter-disciplinary activities, it comes with unique challenges.

Another consideration once the model is in production is infrastructure and maintenance. Using appropriate infrastructure to work with your customers, end users, or platform development team can be a challenge as you can’t just push a jupyter notebook into production.

If your model becomes a success, you then need to work out how to scale it. Planning scalability into models from the outset takes prior planning and no scope drift. Often projects result in a model rebuild at some point to allow for scalability at a later date.


Conclusion

The answer to the title question, "should you use off the shelf or develop a bespoke machine learning model?" largely depends on what you are doing, the company your work for, and the resources available to you.

If it is a greenfield project you might want to develop the platform using an OTS service, if you can find one that fits your use case well enough. If you have an existing platform that you want to trial a machine learning model to automate something, then try off the shelf first. The only reason with a new project to develop something bespoke is if you can’t find an off the shelf service that can be used as a proof of concept, or you have no deadlines and plenty of resources available to you.

If you work for a start-up, an OTS service can cut costs and provide a proof of concept early on to get customers and investors interested, this is especially true if you lack any data to train your model on. Then later on you can get the resources and data to work on a bespoke model. If you work for an enterprise company you might have the time, resources, existing datasets and buy in internally to skip straight to a bespoke model that you can develop over some time.

The main case where you will want to jump to bespoke models is if you are developing a model that is truly novel and cannot make use of any existing off the shelf services. A caveat here is that this assumes a very binary view of the problem and there are many services that support the creation of machine learning systems such as AutoML and ML Ops that have not been taken into consideration.

As a final point, I would encourage you to evaluate any upcoming projects and see if you could speed things up with an initial rollout of an OTS model, especially if it keeps your managers happy, resulting in uninterrupted time to work on the novel bespoke model.


Related Articles