Piccolo 4B (agentscope-ai)v1.7 · 10 s rule
Data: JevBench v1.8.2, published 2026-10-30 · method v1.7 (10 s rule), measured under the rule
Capability 73.6 · Not ranked in the current release
Measured on the 27 Oct 2026 draw, which is not yet equated to the reference scale; stronger anchor runs for that draw are being measured. Shown unranked until then.
Last measured: 2026-10-30 · measurement release: v1.8.2
Apache-2.0 (weights, HF LICENSE + card); FlexDM-derived layout MIT notice retained
intelligence
59.2
calibration
87.9
speed
88.1
cost
74.5
Composite (secondary)
75.4 · Not ranked on this board
Cost and measurement conditions
$0.0071 per 1,000 decisions (estimated)
GPU-time estimate from v1.8.0: one monthly-rented RTX PRO 6000 96 GB, EUR 999/month / 730 hours, ECB USD 1.1133/EUR dated 28 Oct 2026, USD 1.5235/hour; average load 70%. Architecture: small qwen, 4B active parameters; throughput 61525 tokens/s (architecture-class calibration, 29 Oct 2026, efficient-engine prefill; capped at 100,000 tokens/s). The row's own scored cost basis: 1,075,415 input + 0 output tokens over 1,489 decisions. Generated tokens are counted at the same throughput as prefill. GPU-time cost USD 0.00709694905064293/1,000; old list-price estimate USD 0.0216671927468099/1,000 (deepinfra:Qwen/Qwen3.5-4B at USD 0.03/M input and USD 0/M output). Applied min(old list-price estimate, GPU-time cost) = USD 0.00709694905064293/1,000. Estimated, not a bill. Reference deepinfra:Qwen/Qwen3.5-4B is frozen at the 25 Sep cut-off and was deprecated afterwards (replaced by Qwen/Qwen3.5-9B); re-score if it changes (I-3). USD 0.00709694905 per 1,000 decisions on the frozen 1,489-item common set of this draw (USD 0.00707726606 over all 1,500 items).
p50 latency: 0.3 s
Self-hosted latency estimate under the shared method (assumption, not measured)
Measured 30 Oct 2026 on an evaluator-owned Verda VM in FIN-03 (Finland, EU), one NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB reported, driver 580.178.04, image 24.04.cuda13.0. The two Piccolo models used the same VM sequentially. A first-party loopback HTTP shim invoked the pinned compiler and layout at fe6a7af0c6eaf363612c90051243b4e8aeacc90e, with local hash-verified full model files, read-only mounts, empty credentials, offline flags and network cut before sealed input. Python 3.12.12, torch 2.14.1, CUDA 13.0, transformers 5.12.1 and tokenizers 0.22.2 were pinned. FLA and causal-conv1d were absent; native Torch fallbacks were used. Public parity compared complete token IDs, candidates, keys, metadata and repetition behavior against the native compiler. Eighteen public parity cases and the independent golden fixture passed. Fresh timed short and 3,599-token probes followed separate warm-ups. Noul, choice, score, 255-choice, Unicode JSON, multi-question and repetition-boundary probes passed. An 84,389-token synthetic overflow returned HTTP 422 with zero usage. These are synthetic observations, not scored percentiles or full-ceiling capacity proof. A separate pre-warm passed. The complete 1,500-request run took 155.18647408485413 s; raw output sha256 0b6d05c374571c75c16086771edc67e1c914e558febb28aed5b650ad91051590 was independently verified. Post-run health was ready; provider closure and shredding of temporary node and local copies were confirmed with matching retained checksums. GPU rental spend is separate from the benchmark's model-cost estimate.
Measured latency over 1,500 decisions: p50 0.094 s · p95 0.157 s
0 decisions over 10 s.
A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.