Tacet Sonata (CodePawl, 144M packed encoder on mmBERT-small, option-marker softmax in one pass, in-process via the author's tacet package)v1.7 · 10 s rule

Data: JevBench v1.8.18, published 2026-11-20 · method v1.7 (10 s rule), re-scored from the recorded per-decision times

Capability 24.2 · rank #243 of 251 on the open-weights board

Last measured: 2026-10-07 · measurement release: v1.6.1

apache-2.0

Base model: jhu-clsp/mmBERT-smallsource

intelligence

2.2

calibration

46.3

speed

79.2

cost

89.0

Composite (secondary)

0.0 · Composite rank #260

Cost and measurement conditions

$0.0023 per 1,000 decisions (estimated)

ESTIMATE (size-class reference; self-hosted open weights, no public tariff). THE SAME CHOICE THE PUBLISHED quyet-1-0-tiny ROW CARRIES FOR THE IDENTICAL mmBERT-small BACKBONE: DeepInfra base-size encoders at USD 0.005 per 1M input, 0 output - deepinfra:thenlper/gte-base, deepinfra:BAAI/bge-base-en-v1.5, deepinfra:intfloat/e5-base-v2 and deepinfra:sentence-transformers/all-mpnet-base-v2 are all (0.005, 0.0) in the frozen 25 Sep snapshot - x this row's own measured usage.input_tokens (the package's packed length, capped at 4,096). Estimated, not charged. OUTPUT TOKENS ARE 0 BY CONSTRUCTION, verified in the package source (tacet/decoding.py usage_of sets output_tokens 0; the readout is a softmax over option [MASK] markers from one encoder pass and nothing is generated), the published 02 Oct rule for classifier read-outs. PRICING REFERENCE NOTE: jhu-clsp/mmBERT-small still has no explicit market-listing decision (pricing_v15.reference() raises), so an UNLISTED_BASE_MODELS entry would require a published pricing update.

Measured p50 latency: 0.265 s (on-CPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)

Ranking and cap use: 0.68 s (est. incl. network (2x + 150 ms))

Self-hosted model: measured p50 0.265 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.68 s. Model authors can ask us to time a hosted endpoint instead (on request).

x2 + 0.15 s (assumption, not measured)

measured 2026-10-07 on the live v1.6.0/v1.6.1 pool (inputs/input-selfhosted-1500.jsonl, sha256 901983ae...), 1,500 of 1,500 answered, 0 invalid. Weights codepawl/tacet-sonata @ 26f447c35881a80020fe557c95ceb8c61bc3fd12, Apache-2.0, public and ungated, loaded from a local snapshot (no network during the run). Runtime tacet==0.3.0 from PyPI, the author's own install line (`pip install "tacet>=0.3"`) in an isolated venv (torch 2.14.1+cpu, transformers 5.19.0). ADAPTER: the author's own PR fstandhartinger/jevbench#197 file tacet_local.py at head 76eb75da, byte-identical (sha256 c63a4fe0...), reviewed before running: it maps answers only (noul P(true) -> yes/no; choice and score probabilities used as returned, score keys '0'..'n-1') and reaches nothing but the Hub. It ran through the UNCHANGED v1.6 driver run_v16.py; the only addition, in a wrapper, registers the file as jevbench.adapters.tacet_local and lists it in IN_PROCESS_MODULES as laya_local is listed - lane verification and the runtime egress guard unchanged. Synthetic smoke gate passed before any sealed item was sent (PREREG-R59.md, receipts/SMOKE-tacet-sonata.json). THE 23 LONG ITEMS ARE TRUNCATED, NOT REFUSED, by the package's own code at its default and documented max_length 4,096 (tacet/packing.py keeps the questions and cuts the state; it refuses only states over 200,000 characters, which no item here reaches): exactly 23 records report input_tokens = 4,096 and they are answered and scored. RUN ON SANDY'S SHARED CPU (12-core Zen 2, AVX2, fp32 as the package runs off-accelerator, 8 threads), load average 6.6-24.2 (median ~14) logged every 60 s: the Speed axis is a DECLARED LOWER BOUND, and the author reports raw p50 0.042 s on an RTX 3060 against this run's 0.265 s. Latency axis uses the standard self-hosted x2 + 0.15 s adjustment. Serial, one decision per call, no retry, no batching, offline, credential-free. add-requests run 59, benchmarkheaven.com/submit ref 716D1957 (bh-submit 07d384eb-fc4e-4536-ba61-888567efdf34)

Measured latency over 1,500 decisions: p50 0.265 s · p95 0.816 s

3 decisions over 10 s (3 of them answered without error, counted as wrong).

A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.

Breakdowns (not part of the score)

Chance-corrected competence (0 = chance, 100 = perfect) by category on S+P+L1+L2+L3. Raw values, not equated; the same cells feed the compare view.

Capability by subject topic

Rules, policy & law0.0n=1616
Coding & software0.0n=382
Math & numbers0.0n=370
Finance & commerce0.0n=223
Support & operations0.0n=160
Everyday language0.0n=145
Safety & security0.0n=109

Use cases (TypeSafe categories)

Model routing0.0n=524
Legal & compliance0.0n=515
Customer support0.0n=240
E-commerce0.0n=119
Insurance claims0.0n=119
Risk assessment0.2n=118
Financial crime0.0n=101
Feature extraction0.0n=99
Lead generation0.0n=95
Recruiting0.0n=94
Knowledge graphs0.0n=94
LLM guardrails0.0n=93
Moderation0.0n=93
Code linting0.0n=90
Search & retrieval0.0n=90
Science0.0n=90
Advertising0.0n=88
Gaming0.0n=86
Demand forecasting0.0n=83

Cells need at least 30 answered items; † fewer than 30 (indicative only; drawn as hollow, lighter points on the compare radars).

JevBench v1.6.43 values (method v1.6, before the 10 s rule)

Capability 24.31 · Composite 0.02 · rank #232 on the all-systems board · #188 open weights

intelligence 2.24 · calibration 46.38 · speed 80.28 · cost 89.01

v1.6 speed basis (124 items): p50 0.231 s · p95 0.690 s

These figures are kept for comparison. The current figures above are re-scored under the 10 s rule from the recorded per-decision times and replace them.

Published source