Aplomb 1 (5.3B decision model on Qwen3.5-4B, trained readout head, self-hosted)v1.7 · 10 s rule

Data: JevBench v1.8.18, published 2026-11-20 · method v1.7 (10 s rule), re-scored from the recorded per-decision times

Capability 67.1 · rank #100 of 251 on the open-weights board

Last measured: 2026-10-06 · measurement release: v1.6.1

EmpirioLabs Model License 1.0

Base model: Qwen/Qwen3.5-4Bsource

intelligence

45.2

calibration

88.9

speed

88.8

cost

72.4

Composite (secondary)

56.0 · Composite rank #114

Cost and measurement conditions

$0.0083 per 1,000 decisions (estimated)

GPU-time estimate from v1.8.0: one monthly-rented RTX PRO 6000 96 GB, EUR 999/month / 730 hours, ECB USD 1.1133/EUR dated 28 Oct 2026, USD 1.5235/hour; average load 70%. Architecture: small qwen, 4B active parameters; throughput 61525 tokens/s (architecture-class calibration, 29 Oct 2026, efficient-engine prefill; capped at 100,000 tokens/s). The row's own scored cost basis: 1,249,558 input + 0 output tokens over 1,477 decisions = 846.010832769 tokens/decision. GPU-time cost USD 0.00831316153405032/1,000; old list-price cost USD 0.0253803249830738/1,000; applied min(old list-price cost, GPU-time cost) = USD 0.00831316153405032/1,000. The GPU-time estimate is lower. Estimated, not a bill. Intelligence, Calibration, measured latency and the common cost-item membership are unchanged. Prior prices remain in the archived v1.7 releases.

Measured p50 latency: 0.093 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)

Ranking and cap use: 0.34 s (est. incl. network (2x + 150 ms))

Self-hosted model: measured p50 0.093 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.34 s. Model authors can ask us to time a hosted endpoint instead (on request).

x2 + 0.15 s (assumption, not measured)

measured 2026-10-06 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...). Weights empiriolabsai/aplomb-1@f8e2dc81f0505db21a3b7a933ea479658e4ebb8f, bf16, 5.3 B, base Qwen/Qwen3.5-4B plus the shipped 1.3 GB audio encoder and the trained DecisionHead; click-through gate accepted under the EmpirioLabs Model License 1.0 section 2.1, which grants evaluation and the publication of benchmark results without asking. Served behind the author's own TypeSafe wire format by a TRANSPORT SHIM ONLY: run_aplomb.main()'s loading block verbatim, every request handed to THEIR run_aplomb.decide() with THEIR edm_config.json temperatures and abstain_bias and debias=True - the configuration the author names as the reference in #195 and in the model card. Their own aplomb/schema.py documents the request schema as wire-compatible with TypeSafe /v1/systemone and their aplomb/readout.py already emits the answer dicts the unchanged typesafe adapter parses, so no answer field is rewritten and no contributor file is edited. STACK: the author's own documented install line, with transformers pinned to 5.17.0 - the version their own audio sidecar was saved with; their loader compares the saved config to the live one and 5.19.0 fails it on the transformers_version string alone. SPEED CAVEAT: their documented install line does not include causal_conv1d, so transformers falls back to a reference PyTorch implementation it itself calls much slower; the Speed axis is measured on the stack the author documents. One H100 80 GB (Lium). Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. add-requests run 55, GitHub #195

Measured latency over 1,500 decisions: p50 0.093 s · p95 0.124 s

5 decisions over 10 s (5 of them answered without error, counted as wrong).

A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.

Breakdowns (not part of the score)

Chance-corrected competence (0 = chance, 100 = perfect) by category on S+P+L1+L2+L3. Raw values, not equated; the same cells feed the compare view.

Capability by subject topic

Rules, policy & law37.2n=1616
Coding & software54.7n=382
Math & numbers0.0n=370
Finance & commerce42.6n=223
Support & operations43.0n=160
Everyday language56.3n=145
Safety & security37.4n=109

Use cases (TypeSafe categories)

Model routing58.9n=524
Legal & compliance52.5n=515
Customer support39.2n=240
E-commerce35.9n=119
Insurance claims33.8n=119
Risk assessment40.6n=118
Financial crime19.7n=101
Feature extraction25.0n=99
Lead generation41.1n=95
Recruiting14.1n=94
Knowledge graphs10.0n=94
LLM guardrails54.3n=93
Moderation27.5n=93
Code linting13.6n=90
Search & retrieval59.2n=90
Science6.1n=90
Advertising29.3n=88
Gaming0.0n=86
Demand forecasting0.0n=83

Cells need at least 30 answered items; † fewer than 30 (indicative only; drawn as hollow, lighter points on the compare radars).

JevBench v1.6.43 values (method v1.6, before the 10 s rule)

Capability 67.52 · Composite 54.56 · rank #61 on the all-systems board · #37 open weights

intelligence 45.82 · calibration 89.22 · speed 88.88 · cost 57.87

v1.6 speed basis (124 items): p50 0.094 s · p95 0.117 s

These figures are kept for comparison. The current figures above are re-scored under the 10 s rule from the recorded per-decision times and replace them.

Published source