Your model gets quantized and compiled before it reaches the device. Gauntlet measures exactly what that changed — and whether the compression or the silicon is to blame.
| placement.fallback_ops | ≤ 0 · 1 | FAIL |
| memory.flash_mb | ≤ 0.50 · 0.0146 | PASS |
| accuracy.answer_flip_rate | ≤ 0.020 · — | N/M |
| latency.p99_ms | ≤ 15.0 · — | N/M |
L2_NORMALIZATION fell to the CPU, and the build was blocked before a board existed. Three limits read N/M — not measured — because target execution wasn't available, and the report says which three and why rather than passing them.
We run the same evaluation set three ways and compare them pairwise.
Your original FP32 model, run deterministically. Ground truth.
The quantized model on reference kernels — the compression mathematics alone, no vendor code.
The compiled artifact on the vendor toolchain, running on your board — in your enclosure, at your ambient. The only place thermal and bandwidth are real.
Compilers silently reassign operations they can't place. The model still returns correct numbers, so every accuracy test passes — while the op runs on a CPU core and quietly eats the latency budget you sized the chip for. We read it straight out of the compiler's own output.
9 operators · 8 accelerated · 1 fell to the CPU. Vela's own words: Unsupported opType — an L2_NORMALIZATION, on Ethos-U55. The build was blocked with exit code 1 before a board existed. Read the full report
We ship you a pinned container. It runs on your bench, against your eval set, on your board — and the only thing that comes back is the report. No upload, no data room, no security review. We're working with a small number of teams shipping neural networks onto embedded silicon.