Gauntlet qualification report

run 20260830T183302800258Z-1607abca · 2026-08-30T18:33:20+00:00 · target litert-host · board host:<redacted>:arm64

1Verdict

QUALIFIED
LimitDeclaredMeasuredStatus
accuracy.absolute≥ 0.350.59PASS
accuracy.delta_vs_golden≤ 0.10.01881PASS
accuracy.answer_flip_rate≤ 0.250.07512PASS
latency.p99_ms≤ 609.453PASS
placement.fallback_ops≤ 00PASS

Unset limits are reported as measured, not passed. NOT_MEASURED rows count as neither pass nor fail.

detect task: a 'flip' is a confident detection (conf >= 0.25) in one column with no same-class IoU >= 0.5 match in the other, either direction

accuracy metric: mAP@0.5 on the eval set (golden 0.6088, target 0.5900)

Redacted for publication: directory paths reduced to filenames and host names removed (model_path, artifact_path, budgets_path, board_id, host.node). Every SHA-256, SDK version, measurement and verdict is unmodified — the hashes, not the paths, are what identify the files this run used.

2Provenance

ItemValue
reference modelyolov8n_fp32.onnx
sha256 1607abcabc7b3e3edf9ff56d3845d0167be7113a9722ca9bd6bd878bfc8f1cc9
compiled artifactyolov8n_int8.tflite
sha256 26d097b558ef6b39fd9b07affe3b75b095e03538610170b389b9f888e1363fb0
eval setsha256 0b448a6934d978d5e278fa7fb59674eaf2e8761f5ff23672c3e9fc0e70d36e5e
calibration setsha256 —
compilernone (pre-quantized .tflite) 2.2.0
boardhost:<redacted>:arm64
SDKsnumpy 2.5.2, onnxruntime 1.24.3, onnx 1.20.1, ai-edge-litert 2.2.0, ethos-u-vela 5.1.0
hostplatform=macOS-26.5.2-arm64-arm-64bit, machine=arm64, python=3.12.7, node=<redacted>
Gauntlet version0.2.0 git e34ca59e27ec28d99f65fe807795138ff8eaaa91-dirty
budgets filebudgets-yolov8n.yaml
sha256 acb9015730e09d6c40ee8a82db66aa392b402d8ef0886db3662abdd067894f5f

3Accuracy deviation — three-column attribution

ColumnRuntimeVersionKernelsExecutedOutput digestNote
goldenonnxruntime1.24.3cpu-fp32 (single-threaded, deterministic)yesc03d930454042159
schemelitert (TFLite)2.2.0builtin-reference-int8yes3ba106022e3dbd29
targetlitert-host2.2.0litert-hostyes716cd14f43894f49
Comparisonnmax |Δ|RMSESNR dBcos minflipsflip rate
Golden vs Scheme (quantization mathematics)320.64480.00471826.2290.997527220.6875
Scheme vs Target (silicon kernels & compiler)320.49860.00251631.6770.999209120.375
Golden vs Target (end-to-end)320.64480.00468626.2890.997605220.6875

Answer-flip rate of the qualified artifact vs Golden: 0.07512 [0.04225, 0.1127] 95% CI. Two models can score identically and still disagree on individual inputs; flips expose what aggregate accuracy hides.

4Operator placement

293operations
293accelerated
0CPU fallback
0unsupported

Every operation was placed on the accelerator. No silent CPU fallback in this build.

Placement reason, all rows: LiteRT host CPU (XNNPACK-optimized kernels)

Full placement map — all 293 operations
#OpLayerPlacementCycles (est)Time µsPrecision
0QUANTIZEQUANTIZEACCELERATED——int8
1TRANSPOSETRANSPOSEACCELERATED——int8
2PADPADACCELERATED——int8
3CONV_2DCONV_2DACCELERATED——int8
4LOGISTICLOGISTICACCELERATED——int8
5MULMULACCELERATED——int8
6PADPADACCELERATED——int8
7CONV_2DCONV_2DACCELERATED——int8
8LOGISTICLOGISTICACCELERATED——int8
9MULMULACCELERATED——int8
10CONV_2DCONV_2DACCELERATED——int8
11LOGISTICLOGISTICACCELERATED——int8
12MULMULACCELERATED——int8
13SLICESLICEACCELERATED——int8
14CONV_2DCONV_2DACCELERATED——int8
15LOGISTICLOGISTICACCELERATED——int8
16MULMULACCELERATED——int8
17CONV_2DCONV_2DACCELERATED——int8
18LOGISTICLOGISTICACCELERATED——int8
19MULMULACCELERATED——int8
20ADDADDACCELERATED——int8
21SLICESLICEACCELERATED——int8
22QUANTIZEQUANTIZEACCELERATED——int8
23QUANTIZEQUANTIZEACCELERATED——int8
24CONCATENATIONCONCATENATIONACCELERATED——int8
25CONV_2DCONV_2DACCELERATED——int8
26LOGISTICLOGISTICACCELERATED——int8
27MULMULACCELERATED——int8
28PADPADACCELERATED——int8
29CONV_2DCONV_2DACCELERATED——int8
30LOGISTICLOGISTICACCELERATED——int8
31MULMULACCELERATED——int8
32CONV_2DCONV_2DACCELERATED——int8
33LOGISTICLOGISTICACCELERATED——int8
34MULMULACCELERATED——int8
35SLICESLICEACCELERATED——int8
36CONV_2DCONV_2DACCELERATED——int8
37LOGISTICLOGISTICACCELERATED——int8
38MULMULACCELERATED——int8
39CONV_2DCONV_2DACCELERATED——int8
40LOGISTICLOGISTICACCELERATED——int8
41MULMULACCELERATED——int8
42ADDADDACCELERATED——int8
43CONV_2DCONV_2DACCELERATED——int8
44LOGISTICLOGISTICACCELERATED——int8
45MULMULACCELERATED——int8
46CONV_2DCONV_2DACCELERATED——int8
47LOGISTICLOGISTICACCELERATED——int8
48MULMULACCELERATED——int8
49ADDADDACCELERATED——int8
50SLICESLICEACCELERATED——int8
51QUANTIZEQUANTIZEACCELERATED——int8
52QUANTIZEQUANTIZEACCELERATED——int8
53QUANTIZEQUANTIZEACCELERATED——int8
54CONCATENATIONCONCATENATIONACCELERATED——int8
55CONV_2DCONV_2DACCELERATED——int8
56LOGISTICLOGISTICACCELERATED——int8
57MULMULACCELERATED——int8
58PADPADACCELERATED——int8
59CONV_2DCONV_2DACCELERATED——int8
60LOGISTICLOGISTICACCELERATED——int8
61MULMULACCELERATED——int8
62CONV_2DCONV_2DACCELERATED——int8
63LOGISTICLOGISTICACCELERATED——int8
64MULMULACCELERATED——int8
65SLICESLICEACCELERATED——int8
66CONV_2DCONV_2DACCELERATED——int8
67LOGISTICLOGISTICACCELERATED——int8
68MULMULACCELERATED——int8
69CONV_2DCONV_2DACCELERATED——int8
70LOGISTICLOGISTICACCELERATED——int8
71MULMULACCELERATED——int8
72ADDADDACCELERATED——int8
73CONV_2DCONV_2DACCELERATED——int8
74LOGISTICLOGISTICACCELERATED——int8
75MULMULACCELERATED——int8
76CONV_2DCONV_2DACCELERATED——int8
77LOGISTICLOGISTICACCELERATED——int8
78MULMULACCELERATED——int8
79ADDADDACCELERATED——int8
80SLICESLICEACCELERATED——int8
81QUANTIZEQUANTIZEACCELERATED——int8
82QUANTIZEQUANTIZEACCELERATED——int8
83QUANTIZEQUANTIZEACCELERATED——int8
84CONCATENATIONCONCATENATIONACCELERATED——int8
85CONV_2DCONV_2DACCELERATED——int8
86LOGISTICLOGISTICACCELERATED——int8
87MULMULACCELERATED——int8
88PADPADACCELERATED——int8
89CONV_2DCONV_2DACCELERATED——int8
90LOGISTICLOGISTICACCELERATED——int8
91MULMULACCELERATED——int8
92CONV_2DCONV_2DACCELERATED——int8
93LOGISTICLOGISTICACCELERATED——int8
94MULMULACCELERATED——int8
95SLICESLICEACCELERATED——int8
96CONV_2DCONV_2DACCELERATED——int8
97LOGISTICLOGISTICACCELERATED——int8
98MULMULACCELERATED——int8
99CONV_2DCONV_2DACCELERATED——int8
100LOGISTICLOGISTICACCELERATED——int8
101MULMULACCELERATED——int8
102ADDADDACCELERATED——int8
103SLICESLICEACCELERATED——int8
104QUANTIZEQUANTIZEACCELERATED——int8
105QUANTIZEQUANTIZEACCELERATED——int8
106CONCATENATIONCONCATENATIONACCELERATED——int8
107CONV_2DCONV_2DACCELERATED——int8
108LOGISTICLOGISTICACCELERATED——int8
109MULMULACCELERATED——int8
110CONV_2DCONV_2DACCELERATED——int8
111LOGISTICLOGISTICACCELERATED——int8
112MULMULACCELERATED——int8
113MAX_POOL_2DMAX_POOL_2DACCELERATED——int8
114MAX_POOL_2DMAX_POOL_2DACCELERATED——int8
115MAX_POOL_2DMAX_POOL_2DACCELERATED——int8
116CONCATENATIONCONCATENATIONACCELERATED——int8
117CONV_2DCONV_2DACCELERATED——int8
118LOGISTICLOGISTICACCELERATED——int8
119MULMULACCELERATED——int8
120RESIZE_NEAREST_NEIGHBORRESIZE_NEAREST_NEIGHBORACCELERATED——int8
121QUANTIZEQUANTIZEACCELERATED——int8
122QUANTIZEQUANTIZEACCELERATED——int8
123CONCATENATIONCONCATENATIONACCELERATED——int8
124CONV_2DCONV_2DACCELERATED——int8
125LOGISTICLOGISTICACCELERATED——int8
126MULMULACCELERATED——int8
127SLICESLICEACCELERATED——int8
128CONV_2DCONV_2DACCELERATED——int8
129LOGISTICLOGISTICACCELERATED——int8
130MULMULACCELERATED——int8
131CONV_2DCONV_2DACCELERATED——int8
132LOGISTICLOGISTICACCELERATED——int8
133MULMULACCELERATED——int8
134SLICESLICEACCELERATED——int8
135QUANTIZEQUANTIZEACCELERATED——int8
136QUANTIZEQUANTIZEACCELERATED——int8
137CONCATENATIONCONCATENATIONACCELERATED——int8
138CONV_2DCONV_2DACCELERATED——int8
139LOGISTICLOGISTICACCELERATED——int8
140MULMULACCELERATED——int8
141RESIZE_NEAREST_NEIGHBORRESIZE_NEAREST_NEIGHBORACCELERATED——int8
142QUANTIZEQUANTIZEACCELERATED——int8
143QUANTIZEQUANTIZEACCELERATED——int8
144CONCATENATIONCONCATENATIONACCELERATED——int8
145CONV_2DCONV_2DACCELERATED——int8
146LOGISTICLOGISTICACCELERATED——int8
147MULMULACCELERATED——int8
148SLICESLICEACCELERATED——int8
149CONV_2DCONV_2DACCELERATED——int8
150LOGISTICLOGISTICACCELERATED——int8
151MULMULACCELERATED——int8
152CONV_2DCONV_2DACCELERATED——int8
153LOGISTICLOGISTICACCELERATED——int8
154MULMULACCELERATED——int8
155SLICESLICEACCELERATED——int8
156QUANTIZEQUANTIZEACCELERATED——int8
157QUANTIZEQUANTIZEACCELERATED——int8
158CONCATENATIONCONCATENATIONACCELERATED——int8
159CONV_2DCONV_2DACCELERATED——int8
160LOGISTICLOGISTICACCELERATED——int8
161MULMULACCELERATED——int8
162PADPADACCELERATED——int8
163CONV_2DCONV_2DACCELERATED——int8
164LOGISTICLOGISTICACCELERATED——int8
165MULMULACCELERATED——int8
166QUANTIZEQUANTIZEACCELERATED——int8
167CONCATENATIONCONCATENATIONACCELERATED——int8
168CONV_2DCONV_2DACCELERATED——int8
169LOGISTICLOGISTICACCELERATED——int8
170MULMULACCELERATED——int8
171SLICESLICEACCELERATED——int8
172CONV_2DCONV_2DACCELERATED——int8
173LOGISTICLOGISTICACCELERATED——int8
174MULMULACCELERATED——int8
175CONV_2DCONV_2DACCELERATED——int8
176LOGISTICLOGISTICACCELERATED——int8
177MULMULACCELERATED——int8
178SLICESLICEACCELERATED——int8
179QUANTIZEQUANTIZEACCELERATED——int8
180QUANTIZEQUANTIZEACCELERATED——int8
181CONCATENATIONCONCATENATIONACCELERATED——int8
182CONV_2DCONV_2DACCELERATED——int8
183LOGISTICLOGISTICACCELERATED——int8
184MULMULACCELERATED——int8
185PADPADACCELERATED——int8
186CONV_2DCONV_2DACCELERATED——int8
187LOGISTICLOGISTICACCELERATED——int8
188MULMULACCELERATED——int8
189QUANTIZEQUANTIZEACCELERATED——int8
190CONCATENATIONCONCATENATIONACCELERATED——int8
191CONV_2DCONV_2DACCELERATED——int8
192LOGISTICLOGISTICACCELERATED——int8
193MULMULACCELERATED——int8
194SLICESLICEACCELERATED——int8
195CONV_2DCONV_2DACCELERATED——int8
196LOGISTICLOGISTICACCELERATED——int8
197MULMULACCELERATED——int8
198CONV_2DCONV_2DACCELERATED——int8
199LOGISTICLOGISTICACCELERATED——int8
200MULMULACCELERATED——int8
201SLICESLICEACCELERATED——int8
202QUANTIZEQUANTIZEACCELERATED——int8
203QUANTIZEQUANTIZEACCELERATED——int8
204CONCATENATIONCONCATENATIONACCELERATED——int8
205CONV_2DCONV_2DACCELERATED——int8
206LOGISTICLOGISTICACCELERATED——int8
207MULMULACCELERATED——int8
208CONV_2DCONV_2DACCELERATED——int8
209LOGISTICLOGISTICACCELERATED——int8
210MULMULACCELERATED——int8
211CONV_2DCONV_2DACCELERATED——int8
212LOGISTICLOGISTICACCELERATED——int8
213MULMULACCELERATED——int8
214CONV_2DCONV_2DACCELERATED——int8
215TRANSPOSETRANSPOSEACCELERATED——int8
216RESHAPERESHAPEACCELERATED——int8
217CONV_2DCONV_2DACCELERATED——int8
218LOGISTICLOGISTICACCELERATED——int8
219MULMULACCELERATED——int8
220CONV_2DCONV_2DACCELERATED——int8
221LOGISTICLOGISTICACCELERATED——int8
222MULMULACCELERATED——int8
223CONV_2DCONV_2DACCELERATED——int8
224TRANSPOSETRANSPOSEACCELERATED——int8
225RESHAPERESHAPEACCELERATED——int8
226CONV_2DCONV_2DACCELERATED——int8
227LOGISTICLOGISTICACCELERATED——int8
228MULMULACCELERATED——int8
229CONV_2DCONV_2DACCELERATED——int8
230LOGISTICLOGISTICACCELERATED——int8
231MULMULACCELERATED——int8
232CONV_2DCONV_2DACCELERATED——int8
233TRANSPOSETRANSPOSEACCELERATED——int8
234RESHAPERESHAPEACCELERATED——int8
235QUANTIZEQUANTIZEACCELERATED——int8
236QUANTIZEQUANTIZEACCELERATED——int8
237QUANTIZEQUANTIZEACCELERATED——int8
238CONCATENATIONCONCATENATIONACCELERATED——int8
239CONV_2DCONV_2DACCELERATED——int8
240LOGISTICLOGISTICACCELERATED——int8
241MULMULACCELERATED——int8
242CONV_2DCONV_2DACCELERATED——int8
243LOGISTICLOGISTICACCELERATED——int8
244MULMULACCELERATED——int8
245CONV_2DCONV_2DACCELERATED——int8
246TRANSPOSETRANSPOSEACCELERATED——int8
247RESHAPERESHAPEACCELERATED——int8
248CONV_2DCONV_2DACCELERATED——int8
249LOGISTICLOGISTICACCELERATED——int8
250MULMULACCELERATED——int8
251CONV_2DCONV_2DACCELERATED——int8
252LOGISTICLOGISTICACCELERATED——int8
253MULMULACCELERATED——int8
254CONV_2DCONV_2DACCELERATED——int8
255TRANSPOSETRANSPOSEACCELERATED——int8
256RESHAPERESHAPEACCELERATED——int8
257CONV_2DCONV_2DACCELERATED——int8
258LOGISTICLOGISTICACCELERATED——int8
259MULMULACCELERATED——int8
260CONV_2DCONV_2DACCELERATED——int8
261LOGISTICLOGISTICACCELERATED——int8
262MULMULACCELERATED——int8
263CONV_2DCONV_2DACCELERATED——int8
264TRANSPOSETRANSPOSEACCELERATED——int8
265RESHAPERESHAPEACCELERATED——int8
266QUANTIZEQUANTIZEACCELERATED——int8
267QUANTIZEQUANTIZEACCELERATED——int8
268QUANTIZEQUANTIZEACCELERATED——int8
269CONCATENATIONCONCATENATIONACCELERATED——int8
270RESHAPERESHAPEACCELERATED——int8
271TRANSPOSETRANSPOSEACCELERATED——int8
272SOFTMAXSOFTMAXACCELERATED——int8
273CONV_2DCONV_2DACCELERATED——int8
274RESHAPERESHAPEACCELERATED——int8
275SLICESLICEACCELERATED——int8
276SLICESLICEACCELERATED——int8
277SUBSUBACCELERATED——int8
278ADDADDACCELERATED——int8
279ADDADDACCELERATED——int8
280MULMULACCELERATED——int8
281SUBSUBACCELERATED——int8
282CONCATENATIONCONCATENATIONACCELERATED——int8
283MULMULACCELERATED——int8
284LOGISTICLOGISTICACCELERATED——int8
285MULMULACCELERATED——int8
286QUANTIZEQUANTIZEACCELERATED——int8
287CONCATENATIONCONCATENATIONACCELERATED——int8
288DEQUANTIZEDEQUANTIZEACCELERATED——int8
289DELEGATEDELEGATEACCELERATED——int8
290DELEGATEDELEGATEACCELERATED——int8
291DELEGATEDELEGATEACCELERATED——int8
292DELEGATEDELEGATEACCELERATED——int8

Placement from compiler output. Per-layer drift inside a fused NPU command stream is not observable; deviation attribution is per-segment, not per-layer.

5Latency

Percentilems (95% bootstrap CI)
p509.264 [9.26, 9.268] 95% CI
p959.38 [9.368, 9.394] 95% CI
p999.453 [9.421, 9.49] 95% CI
mean9.172 [9.148, 9.193] 95% CI

960 measured inferences; 3 warm-up passes discarded; 0 retries.

7.579 ms11.15 ms

6Memory — capacity

QuantityMeasured
peak RAM— MB
flash footprint— MB

7Memory — bandwidth

Not measured — EMC/DDR counters are available on boards exposing them (e.g. Jetson via tegrastats).

8Thermal

Not measured (no sustained-load soak on this run).

9Worst offenders

Inputmax |Δ|argmax refargmax artifactFlipColumns
0000000000920.620861896238golden vs target
0000000000300.576862706129golden vs target
0000000000780.509262956294golden vs target
0000000001360.48346189479golden vs target
0000000000360.450583708320golden vs target
0000000000810.4473999519golden vs target
0000000000340.434862766292golden vs target
0000000000890.427861896188golden vs target
0000000000740.4161896277golden vs target
0000000001440.3768519399golden vs target

The specific inputs with the largest deviation, for reproduction.

10Regression diff

baseline run 20260829T235659473087Z-1607abca · 2026-08-29T23:57:17+00:00 · artifact sha256 26d097b558ef6b39…

MeasurementBaselineCurrentΔΔ %
accuracy.absolute0.590.5900
accuracy.answer_flip_rate0.075120.0751200
accuracy.delta_vs_golden0.018810.0188100
latency.mean_ms9.3519.172-0.1796-1.92
latency.p50_ms9.3779.264-0.1128-1.2
latency.p95_ms9.4629.38-0.0821-0.868
latency.p99_ms9.5079.453-0.05389-0.567
placement.fallback_ops000—
placement.total_ops29329300
placement.unsupported_ops000—