Aleria Judge 1.1.1 (ALERIA-ML, inference technique on stock Gemma 4 31B, author server, self-hosted)v1.7 · 10 s rule

Composite (secondary)

31.2 · Composite rank #78

Cost and measurement conditions

$0.138 per 1,000 decisions (estimated)

ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 a8a05890ddab2d39d1a1bd543f2b5c83842bdc70aa54766788bc89b7a8a448d7): base string google/gemma-4-31B-it (the stock weights the technique runs on; the author's own provenance file pins the same revision) priced as google/gemma-4-31b-it USD 0.09/M input, USD 0.34/M output. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the author server's own usage block, kept unchanged: input_tokens is the SUM of the prompt lengths of every branch the server scored for the item, not the length of one prompt (the server runs several prompts per question: both option orders, the nine-digit rating branches of a two-option or noul question, and a gated second pass of further orders on a question with 3+ options when the first two orders disagree). That is why the total is high: 5,783,944 input tokens over the 1,500 answered items. output_tokens = 0 (prefill and readout of the label tokens, nothing generated). The offline count through a first-party transcription of the author's prompt builder and the pinned tokenizer predicted the server's numbers EXACTLY, item by item: 1,368 items equal the first-pass count and 132 equal first pass plus gated extra, 0 mismatches. K/V-cache reuse inside the server (cached_tokens) is not recorded in the raw rows, so no cache discount is reflected. There were 0 refusals. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.137802 per 1,000 decisions on the common basis (USD 0.347037 over all 1,500 items) is 2.133x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 and v1.6.24 imajev-9b precedent) and falls outside the official-caps board views by construction. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them); their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence.

x2 + 0.15 s (assumption, not measured)

Evaluator-owned rented Verda GPU VM, one NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (driver 580.178.04), FIN-03 (Finland, EU), pay-as-you-go non-spot, instance type 1RTXPRO6000.30V at USD 2.535/h. The author's own `python -m aleria_judge` from the pinned commit on 127.0.0.1:30104 inside a network-none Docker container with the cut proven from inside before the input was read (loopback only, no default route, DNS and TCP fail), read-only root, tmpfs scratch, read-only weights and task input, empty credential placeholders (checked by name in image and container), HF offline flags, trace off, one request at a time. Timed over loopback on the VM. Measured 22 Oct 2026 (VM created 07:57Z, removed 08:23Z) on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720..., checked locally, on the node and inside the container); determinism 50/50 bit-identical; node input shredded and VM and volume destroyed afterwards (teardown confirmed).

Model and serving disclosure

The author's own `aleria_judge` package (ALERIA-ML/aleria-judge, tag v1.1.1 = commit ed2fcb1ac7bef5ef5e392a971dfe4f39ee53080d, tree e7caf1e6182e22ed58e90e749a750a1bbc4e135b, Apache-2.0, 62 files, 60 text files hash-verified by git blob oid; `main` HEAD is the same commit), unmodified: our harness image adds a second layer on top of the author's own Dockerfile (Dockerfile + aleria_judge/**, built `--network none --pull=false`; its `RUN verify_runtime()` step checks the TensorRT-LLM sources), started as the author's `python -m aleria_judge` (the Dockerfile CMD) with the author's own defaults: `JUDGE_ENGINE=lean` (own Triton/FlashInfer token-tree forward, BF16), effort `high` (no options.effort sent), vision tower loaded (`JUDGE_VISION=true`, 560 soft tokens), `JUDGE_CPU_THREADS=8`, plus `JUDGE_MODEL_PATH=/model`, `JUDGE_HOST=127.0.0.1`, `JUDGE_TRACE=false` and `JUDGE_CONTEXT_LENGTH=131072` (the author's submitted 1.1.1 default for the lean engine, config.py:21-24; /health reported differs_from_evaluated {} before and after the sealed run). The author's compose file was not used. Static source review PASS WITH FINDINGS, no blockers (no subprocess, eval/exec, pickle, torch.load, sockets or HTTP clients in the author package; no trust_remote_code; no rule keyed to item text or benchmark names; item text can leave the process only through /v1/trace, which was switched off). DISCLOSED CODE PATTERNS, cleared by the benchmark owner (21 Oct 2026; author code on a rented Verda pod only, network none): (1) the author's default start downloads the 62.6 GB checkpoint from the Hub (aleria_judge/checkpoint.py:98, probe at :90; the compose file makes this the default) - never reached, because JUDGE_MODEL_PATH=/model returns at checkpoint.py:80-83 with a pre-staged, sha256-verified checkpoint, HF offline flags, empty credentials and a network-none container; (2) the author's compose file uses network_mode: host and passes HF_TOKEN through (docker-compose.yml:28,50) - not used; (3) FlashInfer/Triton JIT-compile kernels at first use (lean_core.py:342, @triton.jit) and the NVIDIA image is not source-audited - run with tmpfs scratch inside the network-none container, after a synthetic smoke in the same container shape. Weights: google/gemma-4-31B-it at 842da3794eaa0b77d5f08bae87a17459d91ff475 (stock, not gated), the 9 serving files (62,578,656,403 bytes) fetched by us and sha256-verified on the host against the pinned revision, mounted read-only at /model (a mount deviation accepted by the benchmark owner) and re-verified inside the container before any item was read. Image: NVIDIA nvcr.io/nvidia/tensorrt-llm/release@sha256:124da3f10b73afbf4242f5c68fb04b48736c61fbe662e715202371fd7f046f43 (tag 1.3.0rc29, pulled by digest on the pod before the container network was cut); run image id sha256:f309f6a9dfc5ee6b20c9601614952fd00cbcbcc7456051e902a4aaadeac74fb9. In-container code check: 29 author files identical to the pinned manifest (0 failures); the stock base image's own tensorrt_llm/ tree (1,120 extra files, 153 of them .py, tree sha256 5551af64...98ed5) is admitted only by that pinned tree hash plus a package-identity check, both OK (this allowance was added by us after two pre-sealed attempts stopped at the code check because the stock image ships that source tree; no item was involved). FIRST MEASUREMENT (found by our own intake: open submission, Decision Index 0.4.1 row 13; it replaces the earlier board entry morj-1.0.1 of the same project, formerly named MORJ; neither was on JevBench before). INFERENCE TECHNIQUE ON STOCK WEIGHTS: no fine-tuning and no generated text; the author reads the model's next-token distribution over option labels for several rotations of the option order and combines them (README). SEVERAL PROMPTS PER QUESTION: see cost_basis; the native usage is the sum over all scored branches and was verified item by item. AUTHOR-DEFAULT CONTEXT LIMIT 131,072 (the author's submitted 1.1.1 default for the lean engine; the author's public Decision Index run used 65,536 and states that requests within 65,536 get identical answers at the higher limit, checked on 200 index requests): the server answers HTTP 422 above the limit, never a truncation; no status proxy was used. On this draw there were 0 refusals and 1,500 HTTP 200 answers. The 15 items whose longest branch is 76k-82k tokens (76,595-81,839) were ANSWERED, but each took 24.9-27.8 s, so under the JevBench 10 s rule they count as failed decisions; at the author's public-run limit of 65,536 the same 15 items would have been refused, so either limit scores them as failed. Every other decision took 0.384 s or less (median 0.13 s). BF16, effort high (the author's default) and the vision tower loaded although the items are text only, as in the author's compose defaults. SERIAL: we sent strictly one request at a time; determinism over the first 50 positions was bit-identical (50/50, 0 argmax flips). The author states that last-bit probability differences are possible on other hardware. METHOD AND TUNING, the author's own statements (docs/method.md, unverified): the Decision Index was also the author's development benchmark and the serving defaults were selected on index runs; the option-count temperature curve (for questions with 3+ options) was fitted on 512 examples from the training splits of BANKING77, CLINC150, WANLI and ARC-Challenge, of which BANKING77, CLINC150 and ARC are also sources of index benchmarks; the author reports that one fit example duplicates a CLINC150 test item up to punctuation and 7 more differ from one by a word or two, and that refitting without them gives the same constants or moves one grid step. We did not check this independently. The prompt layout comes from Mapika/decider, the question list from wcole3/jev-brain. SELF-REPORTED, unverified: public Decision Index 0.3 score 63.43 on H100 PCIe 80 GB and 63.46 on RTX PRO 6000, median / p80 / mean latency 119 / 403 / 320 ms on RTX PRO 6000 (README); the Decision Index lab reports 99.90% decision agreement with the submitted rows on 831 rows (bar 98.5%). HARDWARE: RTX PRO 6000 96 GB, the card the author validated on; latency reflects that card. 10 s RULE: 15 of 1,500 decisions exceeded 10 s (max 27.79 s); no other decision was near the limit.