FrontiersMind Lumma-fev-27Bv1.7 · 10 s rule

Composite (secondary)

5.3 · Composite rank #164

Cost and measurement conditions

$0.544 per 1,000 decisions (estimated)

ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 aa2afcd5a58d0c4cc0e3d6867d590c70b6ac58e971410c27faf9628f97bcf8b7, re-frozen on 22 Oct 2026 with launch-list keys added and nothing else changed; the rates for this base string are identical in the earlier module 81e3578f): base string Qwen/Qwen3.8-27B (the card's and decision_config.json's base_model) priced as qwen/qwen3.8-27b USD 0.42/M input, USD 3.0/M output. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the usage block of the author's own server: input_tokens = tokens attended for the one question, incl. system prompt, state, options and chat template (readout.py:177); output_tokens = the token count of the JSON text of the returned answers (readout.py:202), a synthetic number because nothing is generated, so it depends on the answer (109 per answered item on average here) and is billed at the output rate, which is most of this row's cost. Our offline count through a first-party transcription of the author's prompt rendering and the pinned tokenizer predicted the server's INPUT numbers EXACTLY, item by item: 763,765 input tokens over the 1,485 answered items, and 15 HTTP 422 refusals at the pre-registered ids and positions (the 15 refused items count 0 tokens). OUTPUT tokens: the server reported 162,504 against our pre-registered planning band of 143,966 to 150,660; the server counted 1 to 18 more tokens than our model of the answer-JSON length on 1,354 of the 1,485 answered items. That prediction miss is ours (our output-length model was wrong), is disclosed here, and changes no score: the cost uses the server's native usage like every other row. OVER THE OFFICIAL COST CAP, DISCLOSED: USD 0.544305 per 1,000 decisions on the common basis (USD 0.538862 over all 1,500 items) is 8.426x the official cap USD 0.064595; the row is ranked (v1.6.16 vllm-sr-d3 and v1.6.24 imajev-9b precedent) and falls outside the official-caps board views by construction. COMMON COST BASIS (v1.6.30): cost per 1,000 decisions is computed on the common 1,485 items of this draw, leaving out the 15 very long items (at least 76,358 input tokens as counted by the rows that answered them); their tokens count for no row, whether the row refused or answered them, and a refused item still counts as a wrong answer for Intelligence.

x2 + 0.15 s (assumption, not measured)

Evaluator-owned rented Verda GPU VM, one NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (97,887 MiB, driver 580.178.04), FIN-03 (Finland, EU), pay-as-you-go non-spot, instance type 1RTXPRO6000.30V at USD 2.549/h, 300 GB disk, host RAM 90,484 MiB. The author's own `python3 -m lumma_fev.server` on 127.0.0.1:8000 inside a docker container with network none (cut proven from inside before any item hash or model use: loopback only, no default route, DNS and TCP fail), read-only root, tmpfs scratch, 16 GB shared memory, read-only weights and code, only /output writable, empty credential variables (names-only check), HF offline flags present in the container configuration (receipt asserted before the cut proof), one request at a time, timed over loopback on the VM. Measured 22 Oct 2026 (VM created 10:31Z; sealed run 10:36:58Z to 10:41:32Z, scored loop 234 s) on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b, checked on the node and inside the container). One pass: 1,500 requests plus a 50-item determinism repeat and a health check. The input was shredded from the node and its absence verified; the VM and its volume were removed by exact id (confirmed 10:42:01Z). Estimated rental cost USD 0.49 for this VM plus USD 0.34 for the earlier first VM that failed before any sealed byte. A first VM (10:19Z to 10:27Z) stopped at the offline-flags proof before any sealed byte was sent and was deleted (see code_revision).

Model and serving disclosure

The author's own serving package, unmodified: PyPI `lumma-fev` 0.2.0 (wheel sha256 ac8e5f9380b3ea3591ae7b2eb075b2ca28d2d362038267548636e91352b19a91 = the hash PyPI publishes; sdist sha256 60a9b79cd91d7e3fbdaf18c9a16573e1273dd9959eb2c1639e9c6a95401e8c87; five Python files, 445 lines, byte-identical between wheel and sdist; uploaded 14 Oct 2026). The Hub repository itself ships no server code (the AutoJev source folder was removed by the authors on 14 Oct). No shim, no status-mapping proxy and no prompt, answer or usage rewrite was used: the author's server speaks the frozen POST /v1/systemone wire and returns usage natively, the refusals are its own HTTP 422, and the frozen TypeSafeAdapter, our first-party pod runner and pod driver were unchanged. Installed on the rented pod only, from local files, `pip install --no-deps --no-index --force-reinstall` of exactly five hash-checked wheels (lumma_fev-0.2.0, transformers-5.17.0 sha256 78ec1ce21579b38dfb83950a0658cd119f87212a2fcfdff478096ce9d6c03801, huggingface_hub-1.33.0 04e434b06e100eddbce9a6e817d72693a7884b10a79bd67ab48080d5c07eb899, fastapi-0.136.3 3d2a69bdf04b7e9f3afa292c3bc7a98816bbfafa10bc9b45f3f3700d2f761620, uvicorn-0.54.0 505bdb0f318731d45f1f712071fc781a8981f6847a31c902c9f5e652d4f67faf) inside a `docker build --network none --pull=false`; nothing is pulled from any index. Base image: stock vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, used only as a CUDA / PyTorch base (torch 2.13.0+cu130, Python 3.12.3, pydantic 2.13.5, httpx 0.28.1, safetensors 0.8.0, tokenizers 0.23.2); vLLM itself is NOT used, the author serves with plain transformers and PyTorch. Derived image sha256:79d8477ca43c484d5f13bd15d85b44b5c4acd3df7b72281e07ab37d794ec62b6. In the container, before any weights, every installed lumma_fev/*.py was compared to the pinned PyPI file, the exact versions above were asserted, and flash-linear-attention, causal-conv1d and the Hub `kernels` package were asserted NOT importable. Started as `python3 -m lumma_fev.server --model /models/lumma-fev-27b/snapshots/2c080fb4b44bbad2a140353393d935e8b7af8bcb --host 127.0.0.1 --port 8000 --name lumma-fev-27b` with HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, USE_HUB_KERNELS=NO and an empty LUMMA_FEV_API_KEY (no --cors, no --api-key, no --device-map). As shipped: bfloat16, sdpa attention, readout temperature 0.93705 (decision_config.json), context limit 8,192 tokens (readout.py:11, not configurable from the command line), one request at a time (the server holds a lock). The optional speed kernels (flash-linear-attention, causal-conv1d) were NOT installed, so transformers' slower PyTorch fallback ran for the gated-delta-net layers; they are unreviewed extra code and a kernel variant would be a separate run. Code gate: cleared by the benchmark owner (22 Oct 2026 10:11Z) on a written list of the disclosed code patterns below. DISCLOSED CODE PATTERNS, none reachable in our configuration: (1) Hub downloads of decision_config.json, readout.safetensors or an image processor (readout.py:22-24, 129-131, 154-155) fire only when --model is not a local folder or a request carries images; we pass a local hash-verified folder and the frozen adapter sends no images; (2) `trust_remote_code=True` (local.py:43) in a fallback reached only when decision_config.json is absent; it is present (sha256 e212eb8cf3eb3759c703a19e0b565e4e06d55e4be96caf96d14b0d3c8de30427, asserted in the container before the cut proof) and the snapshot has no .py file and no auto_map; (3) transformers' optional Hub-kernel loaders in the Qwen3.5 model code (modeling_qwen3_5.py:217, 249-300, 437, 500, 840), inactive because USE_HUB_KERNELS=NO and the `kernels` package is not installed; (4) one environment read (LUMMA_FEV_API_KEY, left empty) and an imported but never used httpx client class (client.py). The authors' code has no subprocess, socket, eval, exec, pickle or torch.load call and no request-body logging. The source review (PASS) and the independent runner review both found nothing that required an owner gate; we asked for the owner's clearance anyway under our foreign-code rule. Reviews: a static source review (PASS), an independent runner review and a delta review (one must: a receipt proving the offline flags inside the scored container before the cut proof, closed and re-checked by us before launch); the sealed-input handling and the rented device were admitted with conditions, and the conditions were met. A first VM (before any sealed byte existed) failed exactly that offline-flags proof, because the flags reached the scored container only through its `env -i` entrypoint and not through the Docker container configuration; we then added the same three flags as `-e` options on `docker run` for the scored container (a one-line runner change checked by us with the static checks and the offline proof, fit and guard tests, not by a further independent review round) and the measurement ran once on a second VM; the first VM was deleted. Weights: FrontiersMind/Lumma-fev-27B at 2c080fb4b44bbad2a140353393d935e8b7af8bcb (weights, config, decision_config.json, readout.safetensors, tokenizer and chat template unchanged since the first upload on 14 Oct 2026; later commits only README, NOTICE and benchmark images), 23 files (11 bfloat16 shards, readout.safetensors, tokenizer.json, 10 small files; 52,193,544,229 bytes; manifest sha256 8411f98f33617fea738d6b6ecb932f6c45bbd214391d3615d4043a5e7d09d590), fetched by us from the commit-pinned Hub revision on the pod before the network was cut, and sha256-verified file by file (shard sha256 = the Hub LFS oid, small files = git blob oid; 'VERIFY OK 23 files') on the host before the smoke; the three benchmark images were not used. FIRST MEASUREMENT (found by our own intake: Decision Index 0.4.1 entry; no outside request for this model and no earlier JevBench measurement). AUTHOR-DEFAULT CONTEXT LIMIT: the author's server accepts at most 8,192 tokens per question including state, options and chat template and never truncates; above that it answers HTTP 422 itself ('a question needs N tokens with its state; the limit is 8192 and input is never truncated'). The 15 items of 76,368-80,526 tokens were refused exactly as pre-registered by id and position and are scored wrong, like the other self-hosted rows on this draw; the largest answered item is 1,864 tokens, nowhere near the limit. No status proxy was used. SERVER IS THE AUTHOR'S OWN: lumma_fev.server from the PyPI package, not the Hub repository (which ships no server code); see code_revision for the exact bytes, the disclosed code patterns and the reviews. SCORE QUESTIONS ARE OUTSIDE THE TRAINED TYPES: the card states the model was trained on noul and choice questions and that score questions are accepted but untested; about a quarter of this draw's items are score questions, answered natively by the same readout (probabilities per level), with nothing added or rewritten by us. OPTIONAL KERNELS OFF: flash-linear-attention and causal-conv1d (recommended by the card for speed) were not installed, so transformers' slower PyTorch fallback ran; kernel variants can change speed and numerical results; numerical parity against an optional-kernel configuration was not tested. SERIAL ONLY: the server handles one request at a time and our driver sent one at a time; determinism over the first 50 positions was bit-identical (50/50, 0 argmax flips, max probability difference 0.0). TRAINING DATA: the card gives only 'about 100K samples' in English, Hindi, Tamil and Kannada and names no datasets; there is no training-data licence table and no contamination or overlap statement in the pinned card (the authors' blog post linked from the card was not read). The Decision Index 0.4.1 row carries no overlap or contamination flag. We ran no overlap check, and no JevBench use or denial is declared, so whether its training data overlaps JevBench sources is unknown to us. LICENCE: Apache-2.0 in the card metadata with a full LICENSE and a NOTICE file; the serving package is Apache-2.0 AND MIT (its prompt and scoring follow AutoJev, MIT). SELF-REPORTED, unverified: the card's three benchmark charts (accuracy against 'TypeSafe Jev 1.13.0' on public sets; not transcribed or checked here) and no JevBench or Decision Index number is claimed by the authors. The Decision Index 0.4.1 (22 Oct 2026 snapshot) states for this model: full (balanced_skill) score 59.97, public 57.15 (its kit rescore equals the submitted claim), same-skills 56.59, new-domains 51.01, decision agreement with the submitted rows 99.25% on 825 rows against a 98.5% bar (private tests run by the lab on one RTX PRO 6000 with the author's serving), latency median 100 ms and p95 1,075 ms, and that questions over its declared 8,192-token limit count as unsupported. 10 s RULE: max measured decision latency 0.48 s over all 1,500 rows (answered items: median 0.14 s, p95 0.25 s); 0 decisions over 10 s. In the synthetic smoke before the run a 8,000-token request took 2.17 s and an exact 8,192-token request 2.22 s, but no answered item is larger than 1,864 tokens. OUTPUT TOKENS: our pre-registered output-token planning band was missed (see cost_basis); every other pre-registered prediction (refusal ids and positions, answered count, input tokens item by item, determinism) was met.