Three specialists and a router

Most language models use one mechanism for everything: stack attention blocks, make them wider, make them deeper. That works when memory is cheap. On a part with 20 KB of SRAM it does not, because the thing that kills you is not arithmetic — it is the state you have to hold while you do it.

Atome LM's block runs three structurally different operations in parallel and lets a tiny router decide, per token, how much of each to use. Three small specialists cost less memory than one big generalist at the same quality. That sentence is the entire architectural argument, and everything below is why it is true.

The three pathways

1 · Local — a depthwise causal convolution, k=5

A short-range filter over the last five positions. Ternary kernel, no bias, O(L·k) per token. This is the pathway that handles the small regularities between adjacent characters: the "q" that is followed by "u", the space that follows a period, the plural "s". It is cheap because it is local, and it is local because most of the signal in byte-level text is.

2 · State — a diagonal state-space model

The long-term memory. It carries information from the start of the sentence forward, and — this is the part that matters on a microcontroller — it is recurrent at inference and O(1) per token in state. It does not grow a cache. Its parameters a, b and c_out are kept in FP32 per channel, because the recurrence is the one place where ternary rounding compounds over the length of the sequence.

3 · Sparse — top-k=4 causal attention

For when a token depends on one specific earlier token rather than on a smooth summary of the past. Ternary Q/K/V projections, softmax over only the top four keys per query. Full attention would give a better answer and would also give you a key-value cache whose size grows with context — which is precisely the thing that puts a transformer over the SRAM wall on this class of part. Four keys is the compromise that keeps the copy mechanism and drops the cache.

The router

A per-token softmax over the three pathway outputs. It is small, it is trained jointly, and it has a useful side effect: its entropy is observable for free at every position. A confident router means the block knows which mechanism the current token needs; a flat router means the input does not look like anything it was trained on. In the v2 production line that signal tracks out-of-domain inputs and correlates with per-token loss. At 60K scale on a single corpus the signal is exposed identically, but its calibration as an uncertainty estimator has not been measured here — so it is a diagnostic, not a guarantee.

Why ternary, and where it stops helping

Every projection weight is one of three values, −α, 0 or +α, with α a single per-tensor scale. Three states carry log₂(3) ≈ 1.58 bits, which is where the phrase "1.58-bit model" comes from. Two consequences, one obvious and one not:

• Size. Packed four trits to the byte in base 3 — the ATOME01 format — the 944,000-parameter model is 271 KB instead of 3.7 MB.
• Arithmetic. Multiplying by −1, 0 or +1 is a sign flip, a skip or a copy. The inner loop of inference becomes additions and sign changes, which is exactly what a Cortex-M core with no floating-point unit is good at.

And the cost, stated plainly: three levels per weight is a real capacity limit. The ternary inductive bias substitutes for capacity at small scale and constrains it at large scale. Measured on TinyStories at 3,000 steps, single seed: at 60,000 parameters the routed-ternary block reaches 6.31 perplexity against 8.12 for a parameter-matched FP32 transformer, and 6.31 against 13.10 at fixed flash. At 944,000 parameters the float model wins by about 11% — 2.54 against 2.87. The architecture is a bet on the sub-1M regime, and outside it the bet loses.

What the engine does not have

The C99 engine is a fixed-shape forward pass and nothing else. No malloc, no socket, no fopen, no operating system call of any kind. The absence is checkable in the symbol table of the compiled object, which is a stronger statement than a privacy policy. It compiles to 2.6–2.8 KB of .text across Cortex-M0, M3, M4, M4F and M7, and it makes no assumption about a floating-point unit, a cache or an MMU.

Related

Questions

Why does Atome LM use three pathways instead of attention alone?

Because the binding constraint on a microcontroller is state, not arithmetic. Full attention needs a key-value cache whose size grows with context, and that is what pushes a transformer over the SRAM wall on a part with 20 KB. A depthwise convolution is O(k) in state, a diagonal state-space model is O(1) per token, and top-k=4 attention keeps the copy mechanism without the cache.

What is the router in Atome LM?

A per-token softmax that decides how much of each pathway's output to use for the current token. Its entropy is observable for free at every position, which makes it useful as an out-of-domain diagnostic — though its calibration as an uncertainty estimator has not been measured at 60K scale.

Are all the weights ternary?

All projections are ternary with a per-tensor scale. The diagonal state-space parameters a, b and c_out stay FP32 per channel, because that is the one place where ternary rounding compounds along the length of the sequence.