Amazon Strands Decider 26B-A4Bv1.7 · 10 s rule
Data: JevBench v1.7.1, published 2026-10-25 · method v1.7 (10 s rule), measured under the rule
Capability 66.7 · rank #41 of 159 on the open-weights board
Last measured: 2026-10-24 · measurement release: v1.7.1
Apache-2.0 checkpoint (model card and LICENSE.md); public, not gated; base google/gemma-4-26B-A4B-it: Apache-2.0 per preregistered model metadata
intelligence
46.7
calibration
86.7
speed
88.0
cost
50.5
Composite (secondary)
54.5 · Composite rank #42
Cost and measurement conditions
$0.045 per 1,000 decisions (estimated)
Estimated base-model market reference, not charged and not an Amazon or adapter-hosting quote. Frozen pricing module aa2afcd5 uses lowercase key google/gemma-4-26b-a4b-it at USD 0.09/M input and USD 0.30/M output. The author's capitalised 26B spelling does not resolve in that module; the lowercase module key was used explicitly. Native usage agrees with offline counts item by item: 734625 input and 1485 output tokens across 1485 answers. The server accounts one output token per answer for pointer readout, with no generated explanation. Common cost basis excludes the same 15 long items for every token-priced row; refusals still count wrong for capability. Cost on the common basis: USD 0.044822727273 per 1000 decisions; official cap USD 0.06459465517241379.
Measured p50 latency: 0.111 s (on-GPU, no network, loopback; not directly comparable with API latency, which we time end to end from our measuring host, network included)
Ranking and cap use: 0.37 s (est. incl. network (2x + 150 ms))
Self-hosted model: measured p50 0.111 s on-GPU, no network (timed on our GPU over loopback). Not directly comparable with API latency, which we time end to end from our measuring host, network included. The ranking and latency cap use the same estimate for every self-hosted row: 2 × measured + 150 ms = 0.37 s. Model authors can ask us to time a hosted endpoint instead (on request).
x2 + 0.15 s (assumption, not measured)
Author's server on evaluator-rented Verda RTX PRO 6000 Blackwell 96 GB, FIN-03 Finland, EU, PAYG non-spot. Measured 24 October 2026, 17:59-18:10 UTC, serial loopback requests inside a network-isolated read-only container with pinned local checkpoint/base cache, offline flags and empty credentials. All 1500 requests completed; shred and exact-instance teardown verified. Approximately USD 0.48 GPU rental is separate from the estimated inference tariff.
Measured latency over 1,500 decisions: p50 0.111 s · p95 0.140 s
0 decisions over 10 s.
A decision that takes longer than 10 seconds (measured time) counts as a wrong answer and as zero calibration for that item: each type's calibration is multiplied by the share of its decisions answered within 10 s. The limit uses the measured time (raw). The displayed adjusted time for self-hosted rows (2 × + 0.15 s) is an assumption and is not used for the limit.
Model and serving disclosure
First measurement of Amazon / AWS Strands Decider, a LoRA r48/alpha96 plus pointer readout head on google/gemma-4-26B-A4B-it at 4d7ae4984b7db7de8f8457170b3f1a419ee76d52. Author's own unmodified strands-labs/strands-decider server at b81d443c07438dc5ed51d0e85f9db184ad65d809, peft 0.21.0, BF16 CUDA default attention; vLLM was not imported. Native choice, noul and score outputs, text only, temperature 0.702122181653982. Strict 4096-token window, no truncation: 1485 answers and 15 native HTTP 422 refusals at the predeclared long positions. Refusals count wrong in the full 1500-item capability denominator. First 50 responses repeated bit-identically. The author reports using the public JevBench set for development; training disclosure lists public Hub datasets, ContractNLI/MuSiQue, synthetic/code rows and Qwen teacher distributions. No overlap or contamination audit was run. Public-set evaluation used an older code revision; parity with it is unproven. Earlier create/cache-verifier failures stopped before held-out input and were never scored; only the successful second VM measurement is used. The cache preparation fix did not change weights, serving code, prompts, refusal set or scoring.