Bobcat 1.3 (Gemma 4 31B)v1.7 · 10 s rule

Composite (secondary)

69.4 · Composite rank #16

Cost and measurement conditions

$0.047 per 1,000 decisions (estimated)

ESTIMATE (*est.), base-model market reference through the frozen pricing_v15 module (sha256 aa2afcd5a58d0c4cc0e3d6867d590c70b6ac58e971410c27faf9628f97bcf8b7, re-frozen 22 Oct 2026): base string google/gemma-4-31B-it (the card's base_model) priced as google/gemma-4-31b-it USD 0.09/M input, USD 0.34/M output. Estimated, not charged, not a quote from the author or any host. Tokens as reported by the usage block of the author's own server: input_tokens = the billed count (state plus question; the author's host template of 83 tokens is excluded and a shared state is counted once; the x-bobcat-processed-tokens header is billed + 83 and is NOT used), output_tokens = 0 on every row (the server never generates text). Our offline count through a first-party transcription of the author's prompt compiler and the pinned tokenizer predicted the server's numbers EXACTLY, item by item: 1,976,831 input tokens over the 1,500 answered items (0 item differences), and 0 refusals as pre-registered (all 1,500 items, including the 15 very long ones of 76,000-82,000 tokens, are inside the author's 98,303-token limit). The 15 very long items count for no row's cost on the common 1,485-item cost basis (v1.6.30). COST ON THE COMMON BASIS: USD 0.046884 per 1,000 decisions on 1,485 items, 0.726 times the official cost cap USD 0.064595; estimated, not charged.

x2 + 0.15 s (assumption, not measured)

Evaluator-owned rented Verda GPU VM, one NVIDIA RTX PRO 6000 Blackwell Server Edition 96 GB (97,887 MiB), FIN-03 (Finland, EU), pay-as-you-go non-spot, instance type 1RTXPRO6000.30V at USD 2.549/h. The authors' flash_server (vLLM engine, online FP8, 127.0.0.1:8000) and the 18 hash-pinned author files run in ONE docker container with network none, served at --max-model-len 98368 with --max-num-seqs 64; the driver sent requests one at a time over loopback on the VM. Network cut proven from inside the container before any item hash or model load (loopback only, no default route, DNS and TCP fail); read-only code, read-only model and input, empty credential variables (names-only proof), HF offline flags. Measured 23 Oct 2026 (VM 06:47-07:12Z; measurement 855 s, total 1,475 s; scored run 673 s) on the frozen v1.6-regular-20261012 self-hosted input (sha256 dfd36720077ef7c41ace367524896507c7f44ce71306fb3e659e8b85069d995b, re-checked inside the container). The input was shredded from the node and its absence verified, and the VM was removed by exact id afterwards (teardown confirmed 07:12Z).

Model and serving disclosure

THE AUTHORS' OWN SERVER, run unmodified: foxl-ai/bobcat at 538169365291d7c46d2c116f8ee09ad9f4b3ac5e (5 commits past the previously reviewed 43387a09, +6,032/-45 lines over 34 files, none in the serving path: the 18 serving modules are byte-identical at both commits). Only those 18 files (src/bobcat/*.py, manifest sha256 71dad08892d1e489b312bcd3fffa8a3bcef529181ad1db16deb9c04610d4ff83) were shipped to the VM, read-only at /code/src, hash-verified; every other module of the repo raises ImportError; no git clone, no pip install of the author's package. Command (the 1.3 card's Quickstart plus --identifiers): python3 -m bobcat.flash_server --engine vllm --model /models/bobcat-1.3 --compiler-model /models/bobcat-1.3/compiler --quantization fp8 --temperature choice=1.007793,noul=0.445449,score=1.023435 --name bobcat-1.3 --release-date 2026-10-11 --max-num-seqs 64 --max-model-len 98368 --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN --identifiers /models/bobcat-1.3/bobcat-identifiers.json --host 127.0.0.1 --port 8000 --local (online FP8 of the BF16 weights, the author's recommended path; no kv_cache_dtype override). Image: official vllm/vllm-openai:v0.30.0@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90 plus ONE hash-pinned wheel (scipy 1.15.3), built offline on the VM (image sha256:42b2ad6131642a63abe73bb9c7fa0690ae47356e1be2bb39988971ef5b00d742); vllm 0.30.0, torch 2.13.0+cu130, transformers 5.17.0, tokenizers 0.23.2. The submitted code was reviewed and cleared for this run because the unchanged serving modules contain dormant hooks (DISCLOSED CODE PATTERNS, none reachable in our setup: a Hub downloader and a model-provided-code exec_module fallback in student_readout.py:21,57-63,334-346 (the model repo has no encoding/ directory and ships a chat template, the container has no network); an env read of BOBCAT_EDGE_SECRET at flash_server.py:87 (empty, as --local allows); boto3 at checkpoints.py:102, hf_hub_download at corpus.py:69, httpx at glm_readout.py:191 and torch.load(weights_only=True) at serve.py:33, all training or command-line paths not reachable from flash_server). The source changes and runner were reviewed before the run, findings were fixed and the fixes rechecked. Weights: sanghwa-na/bobcat-1.3 at 4f25c70ece312ec4e74c47f1c5264cb2aeea956a (Apache-2.0, public, not gated, 22 files, BF16 62,546,338,248 weight bytes; the weight average of three merged rank-16 LoRA arms on google/gemma-4-31B-it), fetched by us from the commit-pinned Hub URLs and sha256-verified file by file (MODEL_MANIFEST.json sha256 882e017a6873c6e312277f39db1c91795f440f79a7dfa00e08258a5b31b694f2). FIRST MEASUREMENT (found by our own intake; Author submission, a submission by the authors; our triage treats it as superseding their Bobcat 1.2 submission author submission; the earlier row bobcat-flash-1.2 is a different model). AUTHOR'S OWN SERVER, on our hardware: the row measures sanghwa-na/bobcat-1.3 served by foxl-ai/bobcat as the card's Quickstart documents, in online FP8, on one RTX PRO 6000 (not the author's B200). 10 s RULE: 15 decisions took longer than 10 s (35-38 s each, max 38.5 s) and are scored wrong under the rule in the v1.7 share view (the frozen v1.6 and v1.7-off views do not apply the 10 s rule); they are exactly the 15 standard long positions (76,000-82,000 tokens), consistent with a cold prefill of the whole state (in the smoke a 78.5k-token request took 34.8 s cold and 0.12 s warm); no sealed item was repeated warm. The long items are answered in full (never truncated, never refused). The other 1,485 decisions were fast (p50 0.069 s, p95 0.129 s; the 99th percentile over all 1,500 is 35.1 s only because of these 15). DETERMINISM: first 50 positions run twice, 48/50 bit-identical, 0 argmax flips, max probability difference 0.0012 (online FP8 server). OPEN PARITY RISK: the 1.3 release manifest binds a student_readout.py (sha256 e5597653...) that was never published; the public file (f8d3ceb8e1f33509..., the only version ever pushed, also bound by the 1.2 manifests) is what the card's Quickstart installs and what we ran, so parity with the author's own 1.3 evaluation is unproven. AUTHOR EXPOSURE (card, self-reported, not verified): the JevBench public 231 was evaluated three times by the authors (unofficial Capability 77.2); the Decision Index public half was used for candidate selection; the Choice and Score temperatures were fitted on LocalLLaMA/typed-decisions even test ids and the Noul temperature on a held-out JevBench-format split. The authors' delta audit code drops training rows that overlap the public 231; we did not audit it, ran no overlap check and cannot rule out other overlap. The fitted temperatures (choice 1.007793, noul 0.445449, score 1.023435) are part of the served configuration. The card's latency numbers (61.4 ms on RTX PRO 6000 for Bobcat 1.2, 28.6 ms on B200 for 1.3) are the author's and do not transfer. TOKEN COUNTING: our counter was validated by one native datum from the earlier bobcat-flash-1.2 run (105) and then by the sealed item-by-item comparison, which matched exactly; the image used tokenizers 0.23.2, the offline counter 0.22.2. No outside request for this model; priced as an estimate from the base model, not from any tariff of the authors.