Cloud AI gets the headlines, but the next wave of intelligence is running on microcontrollers with kilobytes of RAM.

Ask most people where AI “lives,” and they'll point at a data center. ChatGPT, Claude, Gemini — enormous models running on racks of GPUs somewhere far away, reachable only through an internet connection. That mental model isn't wrong. It's just incomplete.
Because while everyone is watching the cloud, a quieter shift is happening much closer to the ground — on devices with kilobytes of RAM, no operating system in the traditional sense, and a current draw measured in microamps. The kind of hardware I work with every day.
I'm an embedded systems engineer. I spend my time writing firmware for ARM Cortex-M microcontrollers, building on Zephyr RTOS, and getting tiny devices to talk over BLE and NB-IoT. And from where I sit, the most interesting AI story of the next decade isn't the model that writes your emails. It's the sensor node that can make a decision without ever phoning home.
This post is my case for why. We'll look at where cloud AI hits a wall, what Edge AI really is, how ordinary microcontrollers are becoming intelligent, and — the part most articles skip — where firmware engineers fit into all of it.

Why Cloud AI Isn't Enough
Cloud AI is extraordinary at what it does. But “run everything in a data center” quietly assumes a lot of things that aren't true out in the physical world. Once you've deployed hardware into a factory, a hospital, or a field, five problems show up fast.

Latency
A round trip to the cloud takes time. The device captures data, packages it, sends it over the network, waits for the server to process it, then waits again for the response. For a chatbot, a few hundred milliseconds is invisible. For a machine that needs to shut down a motor the instant it detects a fault, that delay is the difference between a warning and a wrecked bearing.
Anything that has to react in real time — vibration analysis, motor control, collision avoidance — can't afford a network round trip. The decision has to happen where the data is born.

Privacy
Sending data to the cloud means sending data somewhere else. For a medical wearable measuring a patient's vitals, or a camera inside someone's home, that isn't a minor detail — it's often a legal and ethical line. Processing sensitive data on the device itself means it never has to leave, which is frequently the only acceptable design.

Internet Dependency
Cloud AI stops working the moment the connection drops. And many of the most valuable embedded deployments live exactly where connectivity is worst: remote industrial sites, farmland, moving vehicles, underground installations. A device that becomes useless without Wi-Fi isn't a product you can ship into those environments.

Reliability
Networks are unreliable by nature. Packets get lost. Latency spikes. Servers go down for maintenance. If your device's core function depends on all of that working perfectly every single time, you've built fragility into the foundation. On-device intelligence keeps working whether the network is having a good day or not.

Cost
Streaming raw sensor data to the cloud around the clock — and paying for inference on every sample — adds up in bandwidth, in cloud compute, and in battery drain from a radio that never sleeps. Across a fleet of thousands of devices, those costs scale in the wrong direction.

KEY TAKEAWAY Cloud AI assumes low latency, constant connectivity, ample power, and no privacy constraints. In the real physical world, those assumptions break — and that gap is exactly what Edge AI fills.

What Is Edge AI?
Let's define the terms cleanly, because they get blurred a lot.

Edge computing means processing data close to where it's generated — on or near the device — instead of shipping everything to a central server. The “edge” is simply the far end of the network, where the sensors and actuators live.
Edge AI is running machine learning models on that edge hardware. Instead of sending data to the intelligence, you bring the intelligence to the data.
On-device inference is the concrete act at the heart of it: a trained model runs locally on the device, turning raw input — a sensor reading, an image, an audio clip — into a useful output like a classification, a prediction, or a detection. No server involved.
Training still usually happens in the cloud, where there's plenty of compute. But once the model is trained and optimized, it's deployed onto the device and runs there, on its own. Training is heavy and occasional; inference is light and constant. Edge AI splits those two jobs and puts each where it belongs.

Embedded Systems + AI
For a long time, “smart” devices weren't smart at all. They were data collectors — read a sensor, ship the bytes to a server, wait for someone else's computer to think. The microcontroller was a courier, not a brain. That's changing, and the reason is simple: the hardware got good enough.

Microcontrollers Are Becoming Intelligent
Modern microcontrollers ship with more flash, more RAM, faster clocks, and — critically — instructions that accelerate the kind of math machine learning models need. You no longer need a GPU to run a small neural network. A well-optimized model runs inference on a Cortex-M class chip in a few milliseconds, drawing almost no power. Which means the courier can now read the letter and decide what to do about it, without mailing it anywhere.

The Hardware Spectrum
Edge AI isn't one class of device — it's a whole ladder, and you pick the rung that fits the job:
● ARM Cortex-M microcontrollers (like the nRF52 and nRF9160 I work with) sit at the low-power, always-on end. Kilobytes of RAM, months on a battery — ideal for sensor-driven classification and anomaly detection running right next to the data.
● ESP32 adds Wi-Fi and Bluetooth to a capable dual-core MCU, making it a popular pick for connected devices that also need a bit of on-device smarts.
● Raspberry Pi steps up to a full Linux system — more memory, more compute, a richer software stack — ideal when you need heavier models or camera vision, at the cost of more power.
● NVIDIA Jetson sits at the top of the edge ladder, packing a GPU for demanding vision and deep-learning workloads on autonomous machines and robots.
The engineering skill is matching the workload to the smallest, cheapest, lowest-power device that can actually do the job. Reaching for a Jetson when a Cortex-M would do is as much a design failure as the reverse.

Where Firmware Engineers Fit In
Here's the part most articles skip, and I want to be blunt about it: Edge AI is a firmware problem as much as a machine learning problem. Getting a model onto a device is the easy part. Getting it to run reliably, in real time, within a tight memory and power budget, alongside everything else the device has to do — that's firmware engineering. Someone has to:
● Write the drivers that pull clean data off the sensor in the first place.
● Preprocess and filter that data (often with DSP) before it ever reaches the model.
● Fit the model into flash and manage the RAM it needs for inference.
● Schedule inference so it doesn't starve the device's other real-time tasks — which is where an RTOS earns its keep.
● Squeeze the power budget so the whole thing still runs for months on a battery.
The ML engineer trains the model. The firmware engineer makes it live in the real world. Those are different jobs — and the second one is exactly where embedded developers already excel.

KEY TAKEAWAY Microcontrollers are now capable enough to run real ML, turning them from data couriers into decision-makers. But deploying that intelligence reliably is a firmware discipline — home turf for embedded engineers.

Real-World Applications
This isn't theoretical. On-device intelligence is already showing up across industries, and the pattern is always the same: a decision that has to be fast, private, or offline, made right at the source.
● Smart cameras detect people, objects, or events on-device and flag only what matters — instead of streaming hours of raw video to a server.
● Industrial automation uses edge inference to spot defects on a line or react to fault conditions in real time, where a network round trip would be far too slow.
● Predictive maintenance listens to a motor's vibration or current signature and flags early signs of failure before the machine breaks — one of the highest-value uses of TinyML today, and one that leans heavily on signal processing.
● Medical devices and diagnostics analyze biosignals locally, keeping sensitive patient data on the device. This is the exact territory of a portable diagnostic project I built — a force-sensor-based system for gait and foot-pressure analysis that extracts center-of-pressure trajectories and runs classification to flag musculoskeletal issues, without shipping raw clinical data anywhere.
● Wearables track activity, heart rhythm, and fall detection on-wrist, where all-day battery life makes cloud dependence a non-starter.
● Autonomous drones navigate and avoid obstacles using on-board vision, because a drone that has to ask a server what to do will hit the wall before the answer arrives.
● Smart agriculture puts soil, moisture, and crop-health sensing in fields with terrible connectivity — the definition of a place where offline intelligence isn't optional.
● Automotive systems run driver-assistance and sensor-fusion tasks locally under hard real-time deadlines, where safety depends on decisions measured in milliseconds.
The common thread: the intelligence has to be where the action is.
KEY TAKEAWAY Every one of these shares a reason to stay off the cloud — speed, privacy, or a flaky connection. Edge AI is the natural fit for any device that has to sense and act in the physical world.

TinyML: Machine Learning on Kilobytes
So how do you fit a neural network into a device with less memory than a single web page? This is where TinyML comes in — the practice of running machine learning on extremely resource-constrained microcontrollers.
TensorFlow Lite for Microcontrollers
The workhorse here is TensorFlow Lite for Microcontrollers (TFLM). It's a tiny inference runtime built to run on bare-metal MCUs — no operating system required, no dynamic memory allocation in its core, small enough to fit in a few kilobytes. You train your model with the full TensorFlow tooling on a real machine, convert it to the lightweight TFLite format, and TFLM runs that converted model on the device. Tools like Edge Impulse wrap this whole pipeline into something far more approachable.

ON-DEVICE INFERENCE WITH TFLM (C++)
`#include "tensorflow/lite/micro/micro_interpreter.h"

// RAM set aside for the model's input, output, and intermediate tensors.
constexpr int kArenaSize = 16 * 1024; // 16 KB - tune to your model
static uint8_t tensor_arena[kArenaSize];

// ... load model, build op resolver, construct interpreter,
// then interpreter.AllocateTensors() ...

// Copy a preprocessed sensor window into the quantized input tensor.
TfLiteTensor *input = interpreter.input(0);
for (int i = 0; i < window_len; i++) {
input->data.int8[i] = quantize(sensor_window[i]);
}

// Run inference entirely on-device - no network, no cloud.
if (interpreter.Invoke() != kTfLiteOk) {
handle_error();
}

// Read the result and act on it locally.
TfLiteTensor *output = interpreter.output(0);
int8_t score = output->data.int8[0];
if (score > ALERT_THRESHOLD) {
trigger_alert();
}`
The tensor arena is a fixed block of RAM you budget up front — there is no malloc to fall back on.

Quantization
A trained model normally stores its weights as 32-bit floating-point numbers. That's precise, but expensive in both memory and compute. Quantization converts those weights — and often the activations — down to 8-bit integers. The payoff is significant:
● Roughly 4x smaller model size.
● Faster inference, because integer math is cheaper than floating-point on most MCUs.
● Lower power, for the same reason.
The trade-off is a small drop in accuracy — usually surprisingly small, and well worth what you get back. Here's what full int8 quantization looks like with the TFLite converter:

POST-TRAINING INT8 QUANTIZATION (PYTHON)
import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]

Full integer (int8) quantization for microcontrollers

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

A few hundred representative samples calibrate the int8 ranges

def representative_dataset():
for sample in calibration_data.take(200):
yield [tf.cast(sample, tf.float32)]

converter.representative_dataset = representative_dataset
tflite_model = converter.convert() # ~4x smaller than the float32 model

with open("model_int8.tflite", "wb") as f:
f.write(tflite_model)
A small representative dataset lets the converter calibrate the integer ranges without retraining.

Memory Constraints
On a microcontroller you're juggling two separate budgets: flash holds the model's weights and your program code, so the whole model has to physically fit; RAM holds the tensor arena — the working memory the model needs during inference. There's no swapping to disk, no virtual memory, no “just add more RAM.” What you have is what you have, and the model has to live inside it alongside everything else the firmware is doing.

Power Efficiency
Here's a counter-intuitive point that reframes the whole thing: on many battery devices, running inference locally uses less energy than transmitting the raw data would. The radio is often the hungriest part of the system. So doing the thinking on-device — and sending only a tiny result, or nothing at all — can extend battery life rather than shorten it.
Why TinyML Matters Now
TinyML is having its moment because three curves finally crossed: microcontrollers got capable enough, the tooling matured enough for everyday engineers to use, and demand for private, offline, real-time intelligence got loud enough. It brings machine learning to the billions of cheap, low-power devices that will never have a GPU or a reliable internet connection — which is most of the devices in the world.
KEY TAKEAWAY TinyML makes ML fit where it seemingly can't — through a lean runtime like TFLM and aggressive quantization. And because the radio is usually the biggest power drain, on-device inference can lengthen battery life instead of draining it.

The Challenges (This Is Still Hard)
I don't want to oversell this. Edge AI is powerful, but deploying it is difficult, and pretending otherwise helps no one. Here's what you're up against.
● RAM limitations. The tensor arena has to fit in a handful of kilobytes, shared with the rest of your firmware. This alone rules out many model architectures.
● Flash storage. Model weights plus application code have to fit in the flash you've got. Bigger model, less room for everything else.
● Real-time constraints. Inference can't hog the CPU. If the device also has to service interrupts, hold a radio link, and read sensors on a deadline, the model has to slot into that schedule without breaking anything — a scheduling problem an RTOS is built to solve.
● Power consumption. Every millijoule counts on a battery device. Inference frequency, clock speed, and sleep strategy all become design parameters you actively manage.
● Model optimization. Getting a model small and fast enough without wrecking its accuracy is an iterative grind of quantization, pruning, and architecture tuning. It's rarely one-and-done.
● Security. An intelligent device in the field is a target. You have to protect the model itself, secure communication, and validate that firmware and data haven't been tampered with — which is why work like secure device-to-device authentication matters as much as the ML does.
None of these are reasons not to do Edge AI. They're the reasons it's real engineering and not a demo.
KEY TAKEAWAY Edge AI lives inside hard constraints — RAM, flash, timing, power, and security all push back at once. Success comes from disciplined engineering across all of them, not from any single clever trick.

Why Every Embedded Engineer Should Learn AI
I want to push back on a fear I hear from other embedded folks: that AI is going to make firmware engineering obsolete. I think it's the opposite. AI needs firmware engineering more than ever. A machine learning model is useless until it runs reliably on real hardware, in real time, within real constraints. That last mile — from a trained model to a shipping product — is built out of exactly the skills embedded engineers already have:
● RTOS knowledge lets you schedule inference alongside a device's other real-time tasks without missing deadlines. Deterministic scheduling is what keeps an intelligent device from choking on its own thinking.
● DSP and signal processing are the unsung heroes of Edge AI. Raw sensor data is noisy and messy. Clean it up, filter it, run an FFT, extract the right features — and a tiny model suddenly outperforms a bigger model fed garbage.
● Embedded software architecture holds the whole system together — modular drivers, sane memory management, safe firmware updates, fault tolerance. The intelligence is just one more module in a system that still has to be robust.
Those first two aren't abstract. Good preprocessing routinely beats a bigger network, and it's just a few lines of DSP running before the model ever sees the data:

LIGHTWEIGHT DSP PREPROCESSING BEFORE INFERENCE (C)

`// Preprocessing before inference: remove DC offset, then normalize.
// Clean features beat a bigger model fed raw, noisy data.
void preprocess(float *x, size_t n)
{
float mean = 0.0f;
for (size_t i = 0; i < n; i++) mean += x[i];
mean /= (float)n;

for (size_t i = 0; i < n; i++)
    x[i] = (x[i] - mean) / VREF;   // center, then scale to [-1, 1]

Enter fullscreen mode Exit fullscreen mode

}`

And scheduling that inference as its own RTOS task is what keeps it from interfering with the radio, the sensors, and the interrupts the device still has to service on time:

RUNNING INFERENCE AS A ZEPHYR RTOS TASK (C)
`#include

define STACK_SIZE 2048

define PRIORITY 5

K_THREAD_STACK_DEFINE(inference_stack, STACK_SIZE);
static struct k_thread inference_thread;

// Inference runs as its own task, so it never starves the
// device's other real-time work (radio, sensors, ISRs).
void inference_task(void *a, void *b, void *c)
{
while (1) {
read_sensor_window(&window);
preprocess(window, WINDOW_LEN); // filter / FFT / normalize
run_inference(window, &result);
k_msleep(100); // yield the CPU between runs
}
}

int main(void)
{
k_thread_create(&inference_thread, inference_stack, STACK_SIZE,
inference_task, NULL, NULL, NULL,
PRIORITY, 0, K_NO_WAIT);
return 0;
}`

AI doesn't replace the firmware engineer. It hands them a powerful new tool to put inside the systems they already know how to build. The engineer who understands both the silicon and the model is the one who gets to architect these devices — and there aren't many of them yet.
KEY TAKEAWAY AI complements firmware engineering, it doesn't replace it. RTOS scheduling, DSP preprocessing, and solid embedded architecture are what turn a trained model into a reliable product — making embedded engineers more valuable, not less.

Conclusion
The headlines will keep going to the giant models in the cloud, and fairly so — they're remarkable. But the more I work with microcontrollers, the more convinced I am that a huge share of AI's real-world impact will happen quietly, on tiny devices, in places with no reliable internet and no room for latency.
Intelligence is moving out of the data center and into the physical world — into the sensor on the machine, the wearable on the wrist, the node in the field. And the people who know how to make silicon do exactly what you intend, reliably and under tight constraints, are the ones who get to build it.
That's an exciting place to be standing as an embedded engineer. The fundamentals we already obsess over — timing, memory, power, reliability — turn out to be the fundamentals Edge AI is built on. We don't have to leave our field to work on the frontier of AI. The frontier moved onto our hardware.
Your turn:

Are you already deploying ML on constrained hardware, or is Edge AI still on your “someday” list? And do you see AI as a threat to firmware work, or the best thing that's happened to it?
I write about embedded systems, firmware, ARM Cortex-M, RTOS, IoT, TinyML, and DSP. If this resonated — or you disagree — let's talk.