I wrote an article titled “Machine Learning with IoT Devices on the Edge” in the July 2018 issue of Microsoft MSDN Magazine. See https://msdn.microsoft.com/en-us/magazine/mt847186.
There’s no standard definitions for any of the terms in my title. Machine learning is some software system that makes a prediction. An IoT device is, well, basically any kind of hardware device but usually something small with limited power and memory. The Edge is anything that’s connected to the Cloud but not actually part of the Cloud.
In my article I used an example of a small device similar to a Raspberry Pi. You want to run ML software on the device. In most cases you can’t install a lot of heavyweight software on an IoT device so you have to somehow get the ML model (the part that just does the prediction, as opposed to the part that learns/creates the model) onto the device.
I describe two approaches. The first is to write custom C++ code for the device. This works reasonably well for simple ML model like a single-hidden-layer neural network. But it doesn’t work well for large complex models.
The second approach is a look ahead to a system under development at Microsoft called ELL (embedded leaning library). It’s kind of like a cross-compiler for ML models that does lots of clever optimization to shrink a standard ML model down to a size feasible for a tiny IoT device.
The moral of the story is that all of these efforts to get AI/ML intelligence into IoT devices, are very early in development. But there are a lot of companies working very hard and things will happen very quickly over the next couple of years.