Julia-1 (Supersonic Labs)
Data: JevBench v1.6.39, published 2026-10-22
Capability 9.9 · rank #137 of 139 on the open-weights board
Last measured: 2026-10-13 · measurement release: v1.6.15
Apache-2.0 (Hugging Face cardData.license and README; the pinned tree contains no LICENSE file)
intelligence
2.4
calibration
17.3
speed
79.4
cost
89.7
Composite (secondary)
0.0 · Composite rank #185
Cost and measurement conditions
$0.0022 per 1,000 decisions (estimated)
ESTIMATE (*est.), size-class reference; self-hosted open weights, no public bookable tariff for these weights. The frozen pricing_v15 path is asserted with ONE authorised, scoped correction: CoS deputy decision #18761 added the declared base jhu-clsp/mmBERT-small to pricing_v15.UNLISTED_BASE_MODELS, exactly as openbmb/MiniCPM5-2B and answerdotai/ModernBERT-base already are, so reference() returns no market reference for it rather than raising and the ancestry can be carried in base_model instead of being dropped to null. The addition changes no other row: it was proved behaviour-identical over every registry entry on the box before it was installed (r108/receipts/PRICING-EQUIV-R108.json) and independently reviewed (r108/receipts/REVIEW-PRICING-R108.json). Florian's standing 24 Sep rule then gives a clearly labelled size-class estimate: the base-size-encoder line the PUBLISHED v1.6 field already carries - USD 0.005 per 1M input and 0.0 per 1M output, the line modernbert-jev-v2, tacet-sonata and watt-flash-0.1 carry there (no row of THIS draw carries it, and lev-350m carries 0.01/0.0, not this line) (four base-size encoders in the dated 25 Sep 2026 DeepInfra snapshot sha256 84eb7926... all price at exactly that) - so no new release-lane pricing line is created. x this row's own measured tokens: usage.input_tokens comes from the author's OWN audit function engine.encoding_info() and usage.output_tokens is 0 by construction (one encoder forward pass read at marker positions, nothing generated). Estimated, not charged. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them). They are the 15 items the Jev reference refused with max_tokens_exceeded, which the other rows that did not answer them refused or failed on; their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence. This row: USD 0.002196 per 1,000 decisions (0.034x the official cost cap USD 0.064595), against USD 0.002174 over all 1,500 items (v1.6.29).
Measured p50 latency: 0.292 s (on-CPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.73 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.292 s on-CPU, no network (timed on our CPU host over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.73 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Sandy, the evaluator's own shared CPU host (AMD Ryzen 5 3600, 6 cores / 12 threads, Zen 2, AVX2), NOT a dedicated machine: other jobs ran beside it and the 1-minute load average was logged every 60 s into logs/CONDITIONS-julia-1.jsonl (11 samples, 10.87-13.78, mean 12.5). The author ships NO server, so the HTTP surface is OURS: r107/src/julia_shim_r107.py, a pure transport that makes no mapping decision, because the author's own julia.typed.predict_typed already speaks JevBench's typed-question shape exactly (choice keyed by the item's own criteria keys, score keyed '0'..'n-1' with score = sum(i*p_i), noul = p(true)). Engine built with the author's own inference-policy.json (max_length 8192, head_length 512, strict_encoding true), torch backend, device cpu, float32 (their julia_config weight_dtype), thread count left at their own default of 4 - all four read back from the run's own /health rather than asserted. Run inside a loopback-only network namespace whose outbound probe failed as required, offline (HF_HUB_OFFLINE=1), credential-free, nice 5, in a systemd scope with MemoryMax=8G and no swap. Speed is therefore a loaded-shared-CPU figure and a lower bound for this model on a dedicated core or a GPU; the standard self-hosted x2 + 0.15 s adjustment is applied on top and the raw p50/p95 are kept. Measured for this release on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720...) under CoS deputy decision #18209, which admitted this row only after an independent source review and a synthetic-only smoke, both of which were met in that order before the first sealed item. DISCLOSED: the author's own strict_encoding policy REFUSES rather than truncates, so 17 of the 1,500 items (15 over their lossless context budget, 2 with an option above their 48-token option contract) are honest item-level misses recorded as HTTP 422 - predicted exactly, class split included, before the run.