Here is a cost nobody puts on a datasheet. A continuous sensing device classifies a sliding window. The window is 128 samples. A new sample arrives, the window advances by one, and the device asks for a decision. A stateless runtime recomputes all 128 samples — including the 127 that have not changed since the last decision.
Do that at 50 Hz and you are doing 50 full convolutions per second where one timestep of work per layer would have sufficed. On a battery-powered device that is not an inefficiency, it is the power budget.
Why a stateless runtime has no choice
This is an interface problem, not an optimisation problem, and that distinction is the whole point. TensorFlow Lite for Microcontrollers, CMSIS-NN and X-CUBE-AI all expose a per-tensor operation: you hand it an input tensor, it hands you an output tensor. There is nowhere to put per-layer history between calls. The API cannot express "here is one new sample, advance your internal state".
So the runtime does the only thing it can: it treats every decision as if it were the first one. No amount of kernel tuning fixes that, because there is nothing wrong with the kernel.
What delta inference does instead
Keep a ring buffer of activations per layer. When a sample arrives, compute one new timestep in layer 1, feed it to layer 2, compute one new timestep there, and so on. The receptive field is already in the rings. The output is not an approximation of the full recomputation — it is the same arithmetic, reorganised, so the decision is bit-identical.
| Cortex-M4 @ 80 MHz | Cycles | ms |
|---|---|---|
| Full window recompute | 7,453,884 | 93.2 |
| CMSIS-NN convolution, per window | 2,556,548 | 32.0 |
| Delta inference, per decision | 93,598 | 1.17 |
27.3× against CMSIS-NN, 79.6× against our own batch path, and the decisions match full recomputation exactly on every window — verified, not sampled.
Where it stops working
Three boundaries, all of them real, and any of them can make delta inference the wrong choice:
- Stride. The advantage exists only while windows overlap. Streaming wins when a decision is needed more often than every 27 samples against CMSIS-NN. At stride 128 — non-overlapping — it costs 1.6× more.
- RAM. The rings are state you did not have before: about 10 KiB (24,732 B bss against 6,192 B). On a part with 20 KB of SRAM, that is half your memory to buy a compute saving you may not need.
- Architecture. It works cleanly for causal convolutions and recurrences. A layer that needs the whole window at once — global pooling before the classifier is the common case — has to be handled separately or it silently breaks the parity.
Why this is not just "an optimisation"
Because you cannot get there from the other side. A faster kernel makes the wasted recomputation cheaper; it does not stop it happening. The saving comes from the shape of the interface, which means it is available to an engine designed for it and unavailable to one that was not — regardless of how good the kernels are. That is worth knowing when you are reading a benchmark that compares kernels.
All cycle counts here are Unicorn 2.1.4 emulation against the ARM DDI 0403E cycle table. 61% of executed instructions are not in that table and default to one cycle, so treat the absolutes as lower bounds; both arms ran on the same harness, so the ratio is the part that transfers. No silicon measurement exists yet, and no Joules-per-decision figure exists at all.
Questions
What is delta inference?
Computing only what changed. Per-layer activation ring buffers hold the receptive field, so one new sample costs one new timestep per layer instead of recomputing the whole window. The decisions are bit-identical to full recomputation, so it is a reorganisation of the same arithmetic rather than an approximation.
Why can't TensorFlow Lite Micro do this?
Because its interface has nowhere to keep per-layer history between calls. It exposes a per-tensor operation: input tensor in, output tensor out. That is an interface limitation, not a kernel-quality one, so no amount of optimisation reaches it.
When is delta inference a bad idea?
When windows do not overlap — at stride 128 it costs 1.6× more than one batch call — when you cannot spare roughly 10 KiB of SRAM for the ring state, or when a layer needs the whole window at once.