baselabs mammoth-1 (27B)v1.7 · 10 s rule

Composite (secondary)

41.1 · Composite rank #61

Cost and measurement conditions

$0.104 per 1,000 decisions (estimated)

ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 aa2afcd5a58d0c4cc0e3d6867d590c70b6ac58e971410c27faf9628f97bcf8b7, re-frozen on 22 Oct 2026 with launch-list keys added and nothing else changed; the rates for this base string are identical in the earlier module 81e3578f): base string Qwen/Qwen3.5-27B (the card's base_model) priced as qwen/qwen3.5-27b USD 0.195/M input, USD 1.56/M output. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the usage block of OneJev's server (create_app, behind our launcher; input_tokens = prefix plus all suffix tokens, incl. system prompt and chat template; output_tokens = number of questions = 1 per answered single-question item, by design, nothing generated). Our offline count through a first-party replica of the engine's pre-GPU path (bl_render.py) and the pinned tokenizer predicted the server's numbers EXACTLY, item by item: 777,323 input tokens and 1,485 output tokens over the 1,485 answered items, and 15 HTTP 422 refusals at the pre-registered ids and positions. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.103633 per 1,000 decisions on the common basis (USD 0.102596 over all 1,500 items) is 1.604x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 and v1.6.24 imajev-9b precedent) and falls outside the official-caps board views by construction. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them); their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence.

x2 + 0.15 s (assumption, not measured)

Evaluator-owned rented Verda GPU VM, one NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (driver 580.178.04), FIN-03 (Finland, EU), pay-as-you-go non-spot, instance type 1RTXPRO6000.30V at USD 2.549/h. The launcher and engine run in a docker container with network none (cut proven from inside before any item hash or model load: loopback only, no default route, DNS and TCP fail), read-only root, tmpfs scratch, read-only model and input, empty credential variables, HF offline flags, one request at a time, timed over loopback on the VM. Peak CUDA memory allocated after the scored run 59,375 MiB of 97,887 MiB. Measured 22 Oct 2026 (row 09:57-10:06Z, scored run 401 s) on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720..., checked on the node and inside the container). This row's input was shredded from the node and its absence verified, and the VM was removed afterwards.

Model and serving disclosure

The authors' own serving runtime: OneJev qev package at 81ce62f1597c91e46767d4d02d6ac2e18534fe94 plus the authors' serve.patch (sha256 f234c125f1d38d82104b9e1f24e75bde51e4401928acfba2fbf5c59ef2fb46b8; 15 qev/ files, manifest sha256 818ca0fc1d26ec6a2369aac210fb1de41d908a9e52b45e25bc6150714f046501), vendored as byte-pinned files (re-verified inside the smoke container: 15 files OK), no git clone, no pip -e, setup.sh never run. It is run through OUR first-party launcher baselabs_launcher.py (sha256 7a0896373a6677034909ffa56f17558cbd9ceaf389a9dafdea59cfc638ce26ed), which builds the authors' qev.engine.DecisionEngine (bfloat16 + fp32 LM head, block-bidirectional attention on the full-attention layers, slot_lm readout, 256 slots, debias 1, no calibration file so T=1, auto fork mode which ran as 'batched', author limits 32,768 / 65,536 tokens) and hands it to OneJev's own create_app on 127.0.0.1:8000. `qev serve` was NOT used: it routes these checkpoints (config has a vision_config) to MMDecisionEngine, which never attaches the block-bidirectional mask; the DecisionEngine is what the authors' own Decision Index runs use. The launcher adds only the authors' own 32BitIndexMath sequential-fork fallback and one read-only /_launcher/info route; no prompt, truncation, calibration, answer or usage change. After the scored run the engine reported noncausal attention with the mask installed and every counted mask build (3,000 of 3,000) carried block ids; the counters show that the mask was built with block ids, not numerical parity with the Decision Index runs or with a causal-mask comparison (neither was run). Code gate: cleared by the benchmark owner (22 Oct 2026 08:42Z); a later addendum covering the attention-mask monkeypatch, the Hub-kernel path and the setup.sh extras was disclosed to the owner afterwards and no fresh confirmation was claimed. DISCLOSED CODE PATTERNS (download/client/kernel paths are inactive; the attention-mask monkeypatch executes offline as described below): setup.sh (git clone, pip install -e, a CUDA extension build that can download a wheel), qev/client.py (urllib HTTP client class, unused), qev/engine.py and mm_engine.py Hub loaders (a local verified snapshot is passed, offline flags set), readout.py torch.load of a head file (checkpoints say slot_lm, no head file), constraints.txt listing wandb and an OpenTelemetry exporter stack (not installed), an in-process monkeypatch of transformers' create_causal_mask to add block ids (readout.py:104-124, runs inside the network-none container), and transformers' optional Hub-kernel path (inactive: the kernels package is not installed). A static source review and two independent runner reviews preceded the run (their one must was closed and re-checked before launch). Image: our own build on the pod (network on, before any sealed byte) from pytorch/pytorch:2.8.0-cuda12.8-cudnn9-devel@sha256:a7103283ea7113e10ae5d014bd2342acebda0bc53164b2f7b1dd6eb7a766bdb6 plus 50 hash-pinned wheels (--require-hashes, lock sha256 bf9fa61b272ce83fe5a2ac4c6db219484f929083d46efe0f0a5252ab969c31da, installed with --no-index --no-deps); torch 2.8.0+cu128, transformers 5.19.0, tokenizers 0.23.3, flash-linear-attention 0.5.2; causal-conv1d is not installed (transformers uses its torch fallback); row image sha256:598f1f31ed5aa27f455caefcc3a509e2140bcaf49cae685db419f8fecd5f4682. Weights: baselabs/mammoth-1 at 03007ab3c4481bf01c71ef63adbbbc4b2093d71a, the 10 serving files (54,733,723,313 bytes, serve manifest sha256 33025dc9...), fetched by us from the commit-pinned Hub URLs and sha256-verified file by file on the host and again in the smoke container; README, engine.py, setup.sh and the other author files were not used. FIRST MEASUREMENT (found by our own intake: Decision Index 0.4.1 entry; the GitHub request for this model came from a third party, not the authors). AUTHOR-DEFAULT CONTEXT LIMITS: the engine allows 32,768 tokens for state plus the longest question (strict >) and 65,536 per request and raises an error above that, which the authors' runtime returns through OneJev create_app as HTTP 422, never a truncation; the authors train at 16,384 tokens. The 15 items of 76,374-80,534 tokens were refused exactly as pre-registered and are scored wrong; the largest answered item is 1,854 tokens, inside the trained length. NOT AN AUTHOR-SHIPPED SERVER: the authors ship no server file; our first-party launcher around their DecisionEngine and OneJev's web app is what was timed (see code_revision). SERIAL ONLY: the server handles one request at a time and our driver sent one at a time; determinism over the first 50 positions was bit-identical (50/50, 0 argmax flips). causal-conv1d is not installed, so transformers' slower torch convolution fallback ran. LICENCE: MIT in the card metadata since 21 Oct 2026 (commit 9d1962f1 changed 'proprietary' to 'mit' and removed the evaluation-only line); there is no LICENSE file, no copyright holder and no carried Qwen attribution, and the Decision Index row still said 'proprietary' on 22 Oct (stale). Weights are identical to the Decision Index-scored revision 9d1962f1 (the earlier 68.87 row was another weight set and is not used). TRAINING DATA: not disclosed (the Decision Index notes it); the Decision Index states that its checks found signs that the model's training data overlapped one private new-domain test, which it does not count for this model, we cannot rule out overlap on other tests (our inference); the authors' technical report says decontamination against the Jev / Artificial Analysis index benchmarks was done (self-reported, not verified here); we ran no overlap check and no JevBench use or denial is declared. Our own public-versus-sealed check on this row: Intelligence 78.04 on the public items against 74.37 on the sealed ones, a gap of 3.67 points, so no penalty applies. SELF-REPORTED, unverified: Decision Index 0.4 score 71.24 and an Artificial Analysis replica score of 65.9 on the card; the DI lab's own Decision Index 0.4.1 full score is 68.44 (public 75.34, decision agreement with the submitted rows 99.85% on 831 rows against a 98.5% bar), DI-measured latency median 220 ms. 10 s RULE: max measured decision latency 5.78 s (one outlier; all other decisions <=0.49 s; median 0.19 s, p95 0.27 s); 0 decisions over 10 s; a 20k-token request took about 5.7 s warm and a 32k-token request about 17 s in the smoke, but no answered item is that large.