Benchmark evaluation
Offline batch evaluation for AIME 2026, HLE, and GPQA Diamond, built on the DeepInfer offline batch API.
Generation and scoring are separate stages. The generation stage submits every request to the engine in a single batch and writes the raw model outputs to disk; the scoring stage reads those outputs and produces the final metrics. Scoring can therefore be repeated (with a different judge, for example) without regenerating.
The reported metric is Pass@1.
| Benchmark | Tasks | Samples per task | How Pass@1 is computed |
|---|---|---|---|
| AIME 2026 | 30 | 32 | mean accuracy over 32 samples per task |
| HLE | 2500 (2158 text, 342 image) | 1 | single-sample accuracy |
| GPQA Diamond | 198 | 8 | mean accuracy over 8 samples per task |
1. Layout
eval/
|-- run_eval.py # generation entry point (offline batch)
|-- score.py # scoring entry point
|-- common/ # shared utilities: paths, JSONL I/O, Pass@k, loader, judge, encoder
|-- aime/ # one directory per benchmark
| |-- run.sh # two-stage driver: inference, then evaluation
| `-- bench.py # prompt construction, answer extraction, scoring
|-- hle/
|-- gpqa/
|-- data/ # datasets (or point DATA_FILE elsewhere)
`-- outputs/ # results
2. Requirements
Complete the backend-specific installation described in the top-level README.md:
./build.sh installs the requirements and builds the vendored operators and the in-tree
DeepInfer extension. Keep DEEPINFER_ACCELERATOR exported when running evaluation, because it
selects the platform build of the extension. No additional build step is needed for evaluation.
Chat prompts are rendered by the encoder distributed with the checkpoint
(<checkpoint>/encoding/encoding.py), which the scripts load at runtime. The
checkpoint directory must contain that file.
3. Data preparation
The evaluation scripts do not download datasets. Each dataset must be obtained from its upstream source and converted to JSONL.
| Benchmark | Source | Fields |
|---|---|---|
| AIME 2026 | MathArena/aime_2026, or the official AIME 2026 I/II problems | task_id, question, answer (an integer string in the range 0-999) |
| HLE | cais/hle (gated; accept the terms on Hugging Face first) | task_id, question, answer, optional category; image questions are described below |
| GPQA Diamond | Idavidrein/gpqa (gated) or the public mirror hendrydong/gpqa_diamond_mc | task_id, question, answer (a single letter, A-D) |
3.1 Record format
Each dataset is a JSONL file with the fields shown above. Place it at the default path
given in Section 3.3, or point DATA_FILE at it.
// AIME 2026
{"task_id": "aime_2026_0", "question": "Patrick started walking ...", "answer": "277"}
// HLE, text question
{"task_id": "hle_0001", "question": "...", "answer": "...", "category": "Math"}
// GPQA Diamond: The question must contain the four choices inline, labelled (A)-(D).
{"task_id": "gpqa_diamond_0", "question": "Two quantum states ...\n\n(A) 10^-11 eV\n(B) 10^-8 eV\n(C) 10^-9 eV\n(D) 10^-4 eV", "answer": "D", "category": "Physics"}
3.2 Image questions
An HLE image question may reference its image in any of the following ways:
// 1. Path to a local file
{"task_id": "hle_0500", "question": "...", "answer": "...", "category": "Biology/Medicine",
"image": "/path/to/500.png"}
// 2. Inline base64
{"task_id": "hle_0501", "question": "...", "answer": "...", "category": "Physics",
"image": "<base64-string>"}
// 3. multi_modal_data, as consumed by the engine (optionally with image dimensions)
{"task_id": "hle_0502", "question": "...<image>", "answer": "...", "category": "Chemistry",
"multi_modal_data": {"image_urls": ["/path/to/502.jpg"], "image_sizes": [[1080, 1078]]}}
The question must contain an <image> placeholder wherever the image belongs;
generation replaces it with the image token the engine understands.
3.3 Locating the data file
Each driver script defines a default DATA_FILE:
| Benchmark | Default DATA_FILE |
|---|---|
| aime | eval/data/aime_2026.jsonl |
| hle | eval/data/hle_test.jsonl |
| gpqa | eval/data/gpqa_diamond.jsonl |
Place the dataset at the corresponding path, or point DATA_FILE at it:
cd <repo-root>
bash eval/aime/run.sh
bash eval/gpqa/run.sh
DATA_FILE=/data/my_aime.jsonl bash eval/aime/run.sh
DATA_FILE=/data/my_hle.jsonl bash eval/hle/run.sh
DATA_FILE=/data/my_gpqa.jsonl bash eval/gpqa/run.sh
4. Running an evaluation
Each driver script runs two stages:
[1/2] inference -> [2/2] evaluate
Defaults can be overridden with environment variables:
EP=8 N_SAMPLE=4 TAG=smoke bash eval/aime/run.sh # smoke test
bash eval/aime/run.sh # full run with the default model
MODEL=/path/to/DeepSeek-V4.1-Flash-1017 bash eval/aime/run.sh # local HF directory
Available overrides: MODEL, EP, N_SAMPLE, REASONING_EFFORT, TAG,
DATA_FILE.
For HLE the sampling configuration is not adjustable through environment
variables: n_sample=1 is set in the driver script, and temperature=1.0 is the
per-benchmark default in run_eval.py. To run multiple samples, edit --n-sample
in the script. For AIME and GPQA the sample count is configurable through
N_SAMPLE, defaulting to 32 and 8 respectively.
Scoring uses an LLM judge service by default; see the next section. A deterministic rule-based judge is also provided.
5. Scoring
Two scoring modes are available. The default is the LLM judge, which is also the protocol used by the reference evaluation.
| Mode | Method | Dependencies | Determinism |
|---|---|---|---|
llm (default) |
A judge model decides whether the extracted answer is equivalent to the reference answer. | A judge service | Subject to the judge model's own sampling |
rule |
The answer is extracted by fixed rules and compared exactly: the integer inside \boxed{} for AIME, the letter following ANSWER: for GPQA. |
None | Fully deterministic |
HLE supports the LLM judge only. More than half of its answers are free-form
text, so exact matching would mark equivalent answers as wrong and systematically
understate the score; score.py therefore rejects --judge rule for HLE.
To use the LLM judge, provide a judge service:
export EVAL_JUDGE_BASE_URL=https://api.deepseek.com
export EVAL_JUDGE_API_KEY=<key>
export EVAL_JUDGE_MODEL=deepseek-v4-flash
bash eval/aime/run.sh
To score without a judge service, uncomment the rule-based command in stage 2/2 of the driver script, or re-score existing output directly; no regeneration is required:
python eval/score.py --out-dir eval/outputs/aime/<TAG> --judge rule
When the LLM judge is used, summary.json additionally reports
rule_vs_llm_agreement, the fraction of rollouts on which both modes agree. On
AIME 2026 the two modes agreed on all 960 rollouts.
6. Outputs
Results are written to eval/outputs/<benchmark>/<TAG>/:
| File | Contents |
|---|---|
summary.json |
Aggregate metrics. Pass@1 is the headline figure; Pass@k, num_tasks, num_rollouts and coverage_complete are also reported. |
scored.jsonl |
One row per rollout: raw model output, extracted answer, verdict. |
raw_rank*.jsonl |
Raw model outputs as written by each data-parallel rank. |
meta.json |
The complete configuration of the run, for reproducibility. |
A Pass@1 figure is only valid when coverage_complete in summary.json is
true; otherwise one or more ranks or samples are missing.
7. Reference results
The following Pass@1 values are reference figures for the V4.1 model under the default configuration described above.
| Benchmark | Pass@1 |
|---|---|
| AIME 2026 | 99.9% |
| GPQA Diamond | 93.4% |
| HLE-Full | 42.6% |
Due to the stochastic nature of LLM generation, repeated runs may differ by a small margin. These figures are therefore indicative rather than exact, and serve as a sanity check that a deployment is configured correctly.