Machine Learning Sensors: Truly Data-Centric AI
A new approach to embedding machine learning intelligence on edge devices
“Paradoxically, data is the most under-valued and de-glamorised aspect of AI” — Google research authors of “Data Cascades in High-Stakes AI.”
“Data is food for AI” — Andrew Ng, UC Berkeley professor and pioneer of the data-centric AI philosophy.
Machine learning has seen a bifurcation towards both smaller and larger models in recent years. Large-scale language models with hundreds of billions of parameters are being released regularly, and, with no signs of performance saturation, we can expect to see this trend continue.
On the flip side, the field of tiny machine learning (TinyML) — deploying machine learning models on resource-constrained microcontrollers — is also starting to take hold. Commercial applications for TinyML already exist, from keyword spotting in smartphones (e.g., “Hey Siri” and “OK Google”) to person detection for controlling intelligent lighting, HVAC, and security systems.
This bifurcation personifies the competing philosophies of compute-centric and data-centric AI. While large-scale models built into the cloud personify the extreme of compute-centric AI, data-centric AI has until now been restricted by the traditional system architecture. Taking this philosophy to its extreme, we need to place the computing power directly where the data is generated. Enter the machine learning sensor.
The ML Sensor
The ML sensor paradigm draws inspiration from Andrej Karpathy’s concept of “Software 2.0” — the departure of the well-defined, rule-based software we are currently familiar with (described as software 1.0) towards more ‘intelligent’ software encoded in a less interpretable manner, such as in the weights of a neural network.
In a similar vein, we can imagine the existing hardware instantiation of embedded ML embodying the “Sensor 1.0” or “Embedded ML 1.0” paradigm — having sensor data passed to the CPU where model processing occurs. By contrast, the ML sensor paradigm embodies “Sensor 2.0” or “Embedded ML 2.0,” where processing occurs on the sensor itself, with only the higher-level and already processed information being sent to the CPU.
Building upon this, an ML sensor could be defined as:
“An ML sensor is a self-contained system that
utilizes on-device machine learning to extract use-
ful information by observing some complex set of
phenomena in the physical world and reports it
through a simple interface to a wider system.”
Given the above definition, this differs from an IoT device in that the device need not be connected to the internet and uses ML on-device in place of cloud processing. This is also different than a standard sensor configured alongside TinyML since the processing occurs directly on the sensor, as shown below.
This distinction may seem paltry and unimportant, but this approach is beneficial for several practical reasons.
Frictionless Interface. The overarching goal of ML sensors is to abstract the ML specifics from the end-user. From a hardware perspective, people should not need to be experts in ML to build devices incorporating it. By raising the level of abstraction, the friction of incorporating AI into existing hardware systems for non-experts is also reduced. If any customization is required, such as tailoring a facial recognition algorithm to recognize only your face and no one else’s, it should not require a user to load a Python environment and perform model training or transfer learning.
Self-contained and modular. The hardware and software are necessarily self-contained and modular in an ML sensor. Imagine going to a store and buying a “person detector” module, in the same way you might purchase an infrared or temperature sensor. This modularity allows them to be easily integrated into electronics without the need for any ML domain expertise.
Privacy and Security. More robust privacy and security guarantees can be provided in ML sensors than in a more traditional workflow since sensitive information need never leave the device. While this is most obviously applied to video and audio data, in principle, this could be applied to any sensor data.
Composability. Often, when engineers design and build a system, they need to combine sensors in a specific way to solve a more specific problem, such as controlling the movement of a vehicle. The modular nature of an ML sensor allows this to be achieved similarly. More importantly, the ML sensors can be combined with traditional, non-ML sensors, as well as other ML sensors, to create augmented application solutions.
As an example of composability, imagine a simple scenario where we want to create a smart door system that only opens the front door to the house’s owner. This system requires (1) a way for the system to know when a person is at the door, (2) a way for the system to know if the person at the door is the owner, and (3) a way to deactivate the door lock.
The first challenge can be resolved using a person detector “chip.” This output can then be combined with a second “chip” that performs facial recognition. For commonly combined applications such as these, the models could be collapsed into a single ML sensor, with the person detection model built and flashed to the device from an open-source training set such as the visual wake words dataset. Similarly, the facial recognition model can be trained using a model customized via few-shot learning. This sensor could be integrated into a standard electronic locking system, with the output from the ML sensor interacting with the locking mechanism.
The benefits of the above approach are (1) the architect did not need to know anything about machine learning to construct the system since the ML aspects were abstracted from the user, (2) we used composability of sensors to create a solution to a more specific problem than just ‘person detection,” and (3) our data did not need to go anywhere, meaning the system was privacy-preserving and poses minimal security risks. Furthermore, from a scalability perspective, it is much easier to configure thousands of smart doors if they are configured with ML sensors, than having to set up the computing environment for each one manually.
Datasheets
Traditional sensors come with a datasheet, such as this one. An analogous datasheet should be provided with an ML sensor. Why? Well, when someone buys a sensor, they mainly want to know whether it will work for their specific application — you would not buy a temperature sensor for your kettle if you knew it would melt below the boiling point of water, would you?
This is no different for an ML sensor. For a person detector, we want to make sure that the sensor would work in the physical environment where we will deploy it. This might require thinking about features that impact the model, such as the lighting environment or the distance a person will be from the sensor. However, it may also require consideration of hardware characteristics such as operating temperature, power consumption, and inferences per second.
Taking from the wisdom of the sensor community, it makes sense for ML sensors to include a data sheet that includes both these hardware and model characteristics. That way, a potential end-user can review whether this makes sense for their application and can also get some assurances on the device's performance. This may also improve trust in the underlying models due to the added transparency, especially if the data is validated by third-party entities and has other certifications or accreditations.
Below is an illustrative example of what an ML sensor datasheet might look like for a person detector module. Note that this includes sections seen in traditional sensor datasheets, as well as additional sections to outline ML model performance, end-to-end performance analysis, and clearly articulate features regarding privacy, ethical, and environmental considerations.
Ecosystem
One could imagine the ML sensor ecosystem as either closed-source, open-source, or some hybrid of the two. There are pros and cons for each of these approaches. Similar to how open-source development for both ML and other software has radically transformed the software ecosystem, an open-source ecosystem surrounding ML sensors would likely accelerate the pace and innovation of ML sensors.
Analogous to large open-source ML projects like TensorFlow and Keras, one can imagine a similar ecosystem for building robust ML sensors for the most ubiquitous applications, like person detection and keyword spotting. This approach would allow crowdsourcing of data and could draw on community expertise while also ensuring the resulting model and device is publicly available, broadening the overall accessibility of ML resources.
However, a truly open approach might limit the utility of ML sensors to only the most ubiquitous applications since there would be little economic incentive for companies to proffer their own solutions. A closed-source solution is also not ideal since it would restrict accessibility via a paywall, and solutions would likely be suboptimal due to limited expertise and access to data.
Perhaps the most salient approach is a semi-transparent one. This way, companies could provide assurances via their ML sensor datasheet and provide sufficient information to users, such as a validation dataset, to probe and validate model claims without releasing the entire intellectual property of the ML sensor, such as model architecture and full training dataset. This approach would provide a balance between incentivizing open-source solutions for the most ubiquitous applications, providing economic incentives for companies to offer their own solutions, and promoting trust between companies and end-users.
Summary
As with any approach, this paradigm is not without issues. Traditional ML concerns still remain, such as model bias, the potential for adversarial attacks, and reduced interpretability and explainability of our device’s functionality. There is also the potential for ML sensors to be exploited for malicious purposes, such as within weaponry or suppressing freedom of speech. That being said, the net impact afforded by increasing the usability of ML in hardware applications will hopefully be a positive one, especially if suitable ethical considerations are made by individuals when designing and developing ML sensors.
In the same way that electrical engineering evolved from electromagnetism and chemical engineering evolved from chemistry, ML sensors mark a natural stepping stone on the journey of ML from the research domain into a fully-fledged (proto-)engineering discipline that is closely intertwined with electrical engineering. While it is not clear how the field will continue to evolve, I anticipate that it will, in some way, incorporate the idea of ML sensors.
As of the time of writing, nobody has constructed a functional ML sensor, but given the utility of this design approach, it is likely just a matter of time. You can read more about this concept in greater depth in our “Machine Learning Sensors” paper posted on ArXiv and learn more about our latest updates at MLSensors.org.
All images contained in this article are by the author.
