Most published TinyML benchmarks are wrong in one of five specific ways, and we have shipped four of the five ourselves. This is the checklist we now run before any number leaves the building, written as the things that went wrong rather than as advice.
1 · Selecting the best epoch on the test set
The single most common failure. You train, you evaluate each epoch on the test set, you keep the best. The resulting number is real in the sense that the model genuinely scored it, and unrepeatable in the sense that it will not do so on the next dataset. You have used the test set as a validation set and it is no longer held out.
We shipped 0.9223 this way. The leak-free number for the same network is 0.9095. The fix: carve a validation set out of the training data, select on that, and score the test set exactly once.
2 · Shuffling a time series
If consecutive rows are correlated — sensor data always is — a random split puts sample t in training and sample t+1 in test. The model does not have to generalise; it has to remember. Accuracy looks superb and collapses in the field.
For anything worn or carried, the split has to be subject-wise: the people in the test set never appear in training. For anything time-ordered it has to be chronological. We have a board of heads whose apparent wins went from four to one when the split was changed from shuffled to chronological. Same models, same data, different split — different verdict.
3 · An under-trained baseline
Comparison is only meaningful at equal budget. If your model trains for 30 epochs and the baseline gets 15, you have measured your patience, not your architecture. The same applies to hyperparameter search: if you swept yours and took the baseline's defaults, the baseline is a straw man.
On our UCI HAR page the opponent is trained at a matched budget. The weaker as-is TFLite configuration scores 0.8958, which would have flattered us by 1.4 points. We publish against the strong one, because the weak one is not the thing anybody would actually deploy.
4 · A hardcoded or published baseline
The worst one, because it looks like a measurement. A constant in a script —
tflm = 0.8945 — that nobody has re-run, or a figure copied from a vendor's
model zoo measured on different data with a different recipe. We found eight instances
of this class in our own codebase in one audit, several of which had already been
reported as verdicts.
The rule we now apply: the verdict logic reads the opponent's number from the result file of a run we executed, and refuses to run at all if that file is absent rather than falling back to a constant.
5 · Reporting the best seed
Train five times with five seeds and the spread on a small model is large. Report the maximum and you have reported a coin flip. We announced a ternary win of +0.7 points from one seed and had to retract it: the five-seed mean was 87.9% against a TFLite Micro mean of 90.97% — a loss, not a win.
The bar: mean ± standard deviation over at least five seeds, and a claimed difference must exceed roughly 2·SE and hold in the large majority of seeds. On our UCI HAR comparison the paired difference is -0.000204 against a 2·SE of 0.017222, which is why we call it a tie instead of a win.
The checklist
- Is the test set scored exactly once, with selection done on a validation set carved from training?
- Is the split subject-wise or chronological, never shuffled, for correlated data?
- Did the baseline get the same epochs, the same tuning effort and the same data?
- Was every number in the comparison produced by a run you executed, with the artifact on disk?
- Is the reported figure a mean over ≥5 seeds with a standard deviation next to it?
- Is the artifact you measured the one you would actually deploy — static quantization, the real C engine, the full test set?
If any answer is no, the honest word is tie, loss or unverified. Saying that up front costs nothing. Announcing a win on Monday and retracting it on Tuesday costs your reader's trust, permanently, and they are right to withdraw it.
Why publish this
Because a benchmark is only worth reading if the person who ran it has told you how it could be wrong. Ours is at /benchmarks/atome-lm-vs-tflite-micro/, and the axis we lose on is in the same table as the ones we win.
Questions
What is a leak-free benchmark?
One where the test set is scored exactly once and model selection happens on a validation set carved out of the training data. Picking the best epoch by test score is the most common leak, and it produces numbers that are real but unrepeatable.
Why can't I shuffle my sensor data before splitting?
Because consecutive samples are correlated, so a shuffled split puts sample t in training and t+1 in test. The model memorises rather than generalises. Use a subject-wise split for anything worn, or a chronological split for anything time-ordered.
How many seeds does a TinyML benchmark need?
At least five, reported as mean plus standard deviation. A claimed difference should exceed roughly two standard errors and hold across the majority of seeds. Reporting the best of five is reporting a coin flip.