JevBench · open source and community-run

Evaluate Jev-class systems on the data that matters to you

JevBench is our free, open-source benchmark for the community. You can run the same harness yourself, contribute new systems and tasks, or ask us to help with a custom evaluation.

Email us about your data Run it yourself on GitHub

Need custom eval on your data?

We can run relevant entrants on your labelled dataset, report accuracy, calibration, latency and cost, and help you reason about integration, routing and self-hosting. The scope depends on your data and requirements, so it starts with a conversation rather than a package or a price list.

What we need

A labelled sample, a clear task definition, and your expected volume and latency requirements. We use your data only for your evaluation, do not publish it, and delete it on request.

What you get

A written report with our recommendation and the raw results, so you can inspect every system and make your own decision.

Free and open source

Do it yourself with JevBench

Clone github.com/fstandhartinger/jevbench, put your labelled decisions in the repository's JSONL task format, and run the harness with the adapter for your system:

git clone https://github.com/fstandhartinger/jevbench.git
cd jevbench
python -m unittest discover -s tests -v

python -m jevbench.cli run --tasks path/to/your-data.jsonl \
  --adapter typesafe --endpoint https://your-endpoint.example \
  --key-env YOUR_API_KEY --model your-model \
  --results RUN/results.jsonl --raw-dir RUN/raw \
  --ledger RUN/ledger.jsonl --cap-usd 15 --manifest RUN/manifest.json

python -m jevbench.cli summarize --tasks path/to/your-data.jsonl \
  --results RUN/results.jsonl --public-export RUN/summary.json

The code is licensed under the MIT licence. API keys stay in environment variables; use --key-env '' for an endpoint that needs no authorization. The repository includes public example tasks and documents the adapters and result format.

Have a Jev-class system or a useful task set? Open an issue or contribute it on GitHub so the benchmark can grow with the community.

Start with an email

Need custom eval on your data? If you would rather work through it with us, tell us about the task, dataset and decision you need to make. Custom evaluations and consulting are arranged by email.

Email florian.standhartinger@gmail.com to discuss an evaluation.

Benchmark Heaven is operated by productivity-boost.com Betriebs UG (haftungsbeschränkt) & Co. KG. Impressum · Privacy · Terms