baselabs elephant-1 (9B)v1.7 · 10 s rule

Composite (secondary)

60.9 · Composite rank #38

Cost and measurement conditions

$0.052 per 1,000 decisions (estimated)

ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 aa2afcd5a58d0c4cc0e3d6867d590c70b6ac58e971410c27faf9628f97bcf8b7, re-frozen on 22 Oct 2026 with launch-list keys added and nothing else changed; the rates for this base string are identical in the earlier module 81e3578f): base string Qwen/Qwen3.5-9B (the card's base_model) priced as qwen/qwen3.5-9b USD 0.10/M input, USD 0.15/M output. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the usage block of OneJev's server (create_app, behind our launcher; input_tokens = prefix plus all suffix tokens, incl. system prompt and chat template; output_tokens = number of questions = 1 per answered single-question item, by design, nothing generated). Our offline count through a first-party replica of the engine's pre-GPU path (bl_render.py) and the pinned tokenizer predicted the server's numbers EXACTLY, item by item: 777,323 input tokens and 1,485 output tokens over the 1,485 answered items, and 15 HTTP 422 refusals at the pre-registered ids and positions. COST ON THE COMMON BASIS: USD 0.052495 per 1,000 decisions (USD 0.051970 over all 1,500 items) is 0.813x the official cap USD 0.064595, inside the cap. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them); their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence.

x2 + 0.15 s (assumption, not measured)

Evaluator-owned rented Verda GPU VM, one NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (driver 580.178.04), FIN-03 (Finland, EU), pay-as-you-go non-spot, instance type 1RTXPRO6000.30V at USD 2.549/h. The launcher and engine run in a docker container with network none (cut proven from inside before any item hash or model load: loopback only, no default route, DNS and TCP fail), read-only root, tmpfs scratch, read-only model and input, empty credential variables, HF offline flags, one request at a time, timed over loopback on the VM. Peak CUDA memory allocated after the scored run 22,898 MiB of 97,887 MiB. Measured 22 Oct 2026 (row 10:27-10:32Z, scored run 198 s) on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720..., checked on the node and inside the container). This is an elephant-only retry on a fresh VM: the first attempt's elephant row, planned on the mammoth-1 VM, failed before any sealed byte was read (the prepared stack image had been pruned by the mammoth cleanup). On this VM the input was shredded from the node and its absence verified, and the VM was removed afterwards.

Model and serving disclosure

The authors' own serving runtime: OneJev qev package at 81ce62f1597c91e46767d4d02d6ac2e18534fe94 plus the authors' serve.patch (sha256 f234c125f1d38d82104b9e1f24e75bde51e4401928acfba2fbf5c59ef2fb46b8; 15 qev/ files, manifest sha256 818ca0fc1d26ec6a2369aac210fb1de41d908a9e52b45e25bc6150714f046501), vendored as byte-pinned files (re-verified inside the smoke container: 15 files OK), no git clone, no pip -e, setup.sh never run. The authors' code bundle (engine.py, setup.sh, constraints.txt, serve.patch, readout.json) is byte-identical to the one of baselabs/mammoth-1; only config.json, generation_config.json and the weights differ. It is run through OUR first-party launcher baselabs_launcher.py (sha256 7a0896373a6677034909ffa56f17558cbd9ceaf389a9dafdea59cfc638ce26ed), which builds the authors' qev.engine.DecisionEngine (bfloat16 + fp32 LM head, block-bidirectional attention on the full-attention layers, slot_lm readout, 256 slots, debias 1, no calibration file so T=1, auto fork mode which ran as 'batched', author limits 32,768 / 65,536 tokens) and hands it to OneJev's own create_app on 127.0.0.1:8000. `qev serve` was NOT used: it routes these checkpoints (config has a vision_config) to MMDecisionEngine, which never attaches the block-bidirectional mask; the DecisionEngine is what the authors' own Decision Index runs use. The launcher adds only the authors' own 32BitIndexMath sequential-fork fallback and one read-only /_launcher/info route; no prompt, truncation, calibration, answer or usage change. After the scored run the engine reported noncausal attention with the mask installed and every counted mask build (3,000 of 3,000) carried block ids; the counters show that the mask was built with block ids, not numerical parity with the Decision Index runs or with a causal-mask comparison (neither was run). Code gate: cleared by the benchmark owner (22 Oct 2026 08:42Z); a later addendum covering the attention-mask monkeypatch, the Hub-kernel path and the setup.sh extras was disclosed to the owner afterwards and no fresh confirmation was claimed. DISCLOSED CODE PATTERNS (download/client/kernel paths are inactive; the attention-mask monkeypatch executes offline as described below): setup.sh (git clone, pip install -e, a CUDA extension build that can download a wheel), qev/client.py (urllib HTTP client class, unused), qev/engine.py and mm_engine.py Hub loaders (a local verified snapshot is passed, offline flags set), readout.py torch.load of a head file (checkpoints say slot_lm, no head file), constraints.txt listing wandb and an OpenTelemetry exporter stack (not installed), an in-process monkeypatch of transformers' create_causal_mask to add block ids (readout.py:104-124, runs inside the network-none container), and transformers' optional Hub-kernel path (inactive: the kernels package is not installed). A static source review and two independent runner reviews preceded the run (their one must was closed and re-checked before launch); the later runner changes for this elephant-only retry (keep the stack image) were checked by us (diff and dry-run), not by a further review round. Image: our own build on the pod (network on, before any sealed byte) from pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel@sha256:a7103283ea7113e10ae5d014bd2342acebda0bc53164b2f7b1dd6eb7a766bdb6 plus 50 hash-pinned wheels (--require-hashes, lock sha256 bf9fa61b272ce83fe5a2ac4c6db219484f929083d46efe0f0a5252ab969c31da, installed with --no-index --no-deps); torch 2.8.0+cu128, transformers 5.19.0, tokenizers 0.23.3, flash-linear-attention 0.5.2; causal-conv1d is not installed (transformers uses its torch fallback); stack image sha256:76a3333125ebdae54475f49994d3e01ea758a0f82c1f85c0ccdf0ebf03e8dbd5, row image sha256:d4f70ee15d359d0c64a093013bf9bab19f4d4e877469c89142df68f39dc00a29. Weights: baselabs/elephant-1 at c54a55c2d362a7e30ff30814d28e39054e954d58 (licence flip commit 5494c13e; weights commit 14537f08, unchanged since), the 8 serving files (18,839,725,537 bytes, serve manifest sha256 c7202744...), fetched by us from the commit-pinned Hub URLs and sha256-verified file by file on the host and again in the smoke container; README, engine.py, setup.sh, constraints.txt, serve.patch and the banner image were not used. FIRST MEASUREMENT (found by our own intake: Decision Index 0.4.1 entry; no outside request for this model). AUTHOR-DEFAULT CONTEXT LIMITS: the engine allows 32,768 tokens for state plus the longest question (strict >) and 65,536 per request and raises an error above that, which the authors' runtime returns through OneJev create_app as HTTP 422, never a truncation; the authors train at 16,384 tokens. The 15 items of 76,374-80,534 tokens were refused exactly as pre-registered and are scored wrong; the largest answered item is 1,854 tokens, inside the trained length. NOT AN AUTHOR-SHIPPED SERVER: the authors ship no server file; our first-party launcher around their DecisionEngine and OneJev's web app is what was timed (see code_revision). SERIAL ONLY: the server handles one request at a time and our driver sent one at a time; determinism over the first 50 positions was bit-identical (50/50, 0 argmax flips). causal-conv1d is not installed, so transformers' slower torch convolution fallback ran. LICENCE: MIT in the card metadata since 21 Oct 2026 (commit 5494c13e changed 'proprietary' to 'mit' and removed the evaluation-only line); there is no LICENSE file, no copyright holder and no carried Qwen attribution, and the Decision Index row still said 'proprietary' on 22 Oct (stale). Weights are identical to the Decision Index-scored checkpoint (14537f08). Two README drafts briefly named Qwen3.5-27B as base model; the pinned card and the config say Qwen3.5-9B. TRAINING DATA: the card gives a mix of 6.21M decision examples (9.5% public decision data) without naming datasets; the Decision Index states that the training includes train splits of Decision Index benchmarks (author-declared), and that its checks found signs that the model's training data overlapped one private new-domain test, which it does not count for this model; we cannot rule out overlap on other tests (our inference); the authors' technical report separately claims zero contamination with the Jev / Artificial Analysis index benchmarks (self-reported, not verified here); we ran no overlap check and no JevBench use or denial is declared, so whether those Decision Index train splits overlap JevBench sources is unknown to us. SELF-REPORTED, unverified: Decision Index 0.4 score 62.65 ('best model under 15B', tied #8 of 208) and an Artificial Analysis replica score of 65.2 on the card; the DI lab's own Decision Index 0.4.1 full score is 60.26 (public 72.24, equal to the claim by the kit rescore; decision agreement with the submitted rows 99.84% on 819 rows against a 98.5% bar), DI-measured latency median 114 ms. 10 s RULE: max measured decision latency 0.30 s (median 0.08 s, p95 0.10 s); 0 decisions over 10 s; in the smoke a 20k-token request took about 1.9 s warm and an exact 32,768-token request about 9.4 s, but no answered item is that large.