DeepInfer/eval/README.md

# Benchmark evaluation

Offline batch evaluation for AIME 2026, HLE, and GPQA Diamond, built on the
DeepInfer offline batch API.

Generation and scoring are separate stages. The generation stage submits every
request to the engine in a single batch and writes the raw model outputs to disk;
the scoring stage reads those outputs and produces the final metrics. Scoring can
therefore be repeated (with a different judge, for example) without regenerating.

The reported metric is **Pass@1**.

| Benchmark | Tasks | Samples per task | How Pass@1 is computed |
| :--- | ---: | ---: | :--- |
| [AIME 2026](aime/run.sh) | 30 | 32 | mean accuracy over 32 samples per task |
| [HLE](hle/run.sh) | 2500 (2158 text, 342 image) | 1 | single-sample accuracy |
| [GPQA Diamond](gpqa/run.sh) | 198 | 8 | mean accuracy over 8 samples per task |

---

## 1. Layout

```
eval/
|-- run_eval.py          # generation entry point (offline batch)
|-- score.py             # scoring entry point
|-- common/              # shared utilities: paths, JSONL I/O, Pass@k, loader, judge, encoder
|-- aime/                # one directory per benchmark
|   |-- run.sh           #   two-stage driver: inference, then evaluation
|   `-- bench.py         #   prompt construction, answer extraction, scoring
|-- hle/
|-- gpqa/
|-- data/                # datasets (or point DATA_FILE elsewhere)
`-- outputs/             # results
```

---

## 2. Requirements

Complete the backend-specific installation described in the top-level [README.md](../README.md):
`./build.sh` installs the requirements and builds the vendored operators and the in-tree
DeepInfer extension. Keep `DEEPINFER_ACCELERATOR` exported when running evaluation, because it
selects the platform build of the extension. No additional build step is needed for evaluation.

Chat prompts are rendered by the encoder distributed with the checkpoint
(`<checkpoint>/encoding/encoding.py`), which the scripts load at runtime. The
checkpoint directory must contain that file.

---

## 3. Data preparation

The evaluation scripts do not download datasets. Each dataset must be obtained
from its upstream source and converted to JSONL.

| Benchmark | Source | Fields |
| :--- | :--- | :--- |
| AIME 2026 | [MathArena/aime_2026](https://huggingface.co/datasets/MathArena/aime_2026), or the official AIME 2026 I/II problems | `task_id`, `question`, `answer` (an integer string in the range 0-999) |
| HLE | [cais/hle](https://huggingface.co/datasets/cais/hle) (gated; accept the terms on Hugging Face first) | `task_id`, `question`, `answer`, optional `category`; image questions are described below |
| GPQA Diamond | [Idavidrein/gpqa](https://huggingface.co/datasets/Idavidrein/gpqa) (gated) or the public mirror [hendrydong/gpqa_diamond_mc](https://huggingface.co/datasets/hendrydong/gpqa_diamond_mc) | `task_id`, `question`, `answer` (a single letter, A-D) |

### 3.1 Record format

Each dataset is a JSONL file with the fields shown above. Place it at the default path
given in Section 3.3, or point `DATA_FILE` at it.

```jsonc
// AIME 2026
{"task_id": "aime_2026_0", "question": "Patrick started walking ...", "answer": "277"}

// HLE, text question
{"task_id": "hle_0001", "question": "...", "answer": "...", "category": "Math"}

// GPQA Diamond: The question must contain the four choices inline, labelled (A)-(D).
{"task_id": "gpqa_diamond_0", "question": "Two quantum states ...\n\n(A) 10^-11 eV\n(B) 10^-8 eV\n(C) 10^-9 eV\n(D) 10^-4 eV", "answer": "D", "category": "Physics"}
```

### 3.2 Image questions

An HLE image question may reference its image in any of the following ways:

```jsonc
// 1. Path to a local file
{"task_id": "hle_0500", "question": "...", "answer": "...", "category": "Biology/Medicine",
 "image": "/path/to/500.png"}

// 2. Inline base64
{"task_id": "hle_0501", "question": "...", "answer": "...", "category": "Physics",
 "image": "<base64-string>"}

// 3. multi_modal_data, as consumed by the engine (optionally with image dimensions)
{"task_id": "hle_0502", "question": "...<image>", "answer": "...", "category": "Chemistry",
 "multi_modal_data": {"image_urls": ["/path/to/502.jpg"], "image_sizes": [[1080, 1078]]}}
```

The question must contain an `<image>` placeholder wherever the image belongs;
generation replaces it with the image token the engine understands.

### 3.3 Locating the data file

Each driver script defines a default `DATA_FILE`:

| Benchmark | Default `DATA_FILE` |
| :--- | :--- |
| aime | `eval/data/aime_2026.jsonl` |
| hle | `eval/data/hle_test.jsonl` |
| gpqa | `eval/data/gpqa_diamond.jsonl` |

Place the dataset at the corresponding path, or point `DATA_FILE` at it:

```bash
cd <repo-root>

bash eval/aime/run.sh
bash eval/gpqa/run.sh

DATA_FILE=/data/my_aime.jsonl bash eval/aime/run.sh
DATA_FILE=/data/my_hle.jsonl  bash eval/hle/run.sh
DATA_FILE=/data/my_gpqa.jsonl bash eval/gpqa/run.sh
```

---

## 4. Running an evaluation

Each driver script runs two stages:

```
[1/2] inference  ->  [2/2] evaluate
```

Defaults can be overridden with environment variables:

```bash
EP=8 N_SAMPLE=4 TAG=smoke bash eval/aime/run.sh            # smoke test
bash eval/aime/run.sh                                      # full run with the default model
MODEL=/path/to/DeepSeek-V4.1-Flash-1017 bash eval/aime/run.sh   # local HF directory
```

Available overrides: `MODEL`, `EP`, `N_SAMPLE`, `REASONING_EFFORT`, `TAG`,
`DATA_FILE`.

For HLE the sampling configuration is not adjustable through environment
variables: `n_sample=1` is set in the driver script, and `temperature=1.0` is the
per-benchmark default in `run_eval.py`. To run multiple samples, edit `--n-sample`
in the script. For AIME and GPQA the sample count is configurable through
`N_SAMPLE`, defaulting to 32 and 8 respectively.

Scoring uses an LLM judge service by default; see the next section. A
deterministic rule-based judge is also provided.

---

## 5. Scoring

Two scoring modes are available. The default is the LLM judge, which is also the
protocol used by the reference evaluation.

| Mode | Method | Dependencies | Determinism |
| :--- | :--- | :--- | :--- |
| `llm` (default) | A judge model decides whether the extracted answer is equivalent to the reference answer. | A judge service | Subject to the judge model's own sampling |
| `rule` | The answer is extracted by fixed rules and compared exactly: the integer inside `\boxed{}` for AIME, the letter following `ANSWER:` for GPQA. | None | Fully deterministic |

HLE supports the LLM judge only. More than half of its answers are free-form
text, so exact matching would mark equivalent answers as wrong and systematically
understate the score; `score.py` therefore rejects `--judge rule` for HLE.

To use the LLM judge, provide a judge service:

```bash
export EVAL_JUDGE_BASE_URL=https://api.deepseek.com
export EVAL_JUDGE_API_KEY=<key>
export EVAL_JUDGE_MODEL=deepseek-v4-flash
bash eval/aime/run.sh
```

To score without a judge service, uncomment the rule-based command in stage 2/2 of
the driver script, or re-score existing output directly; no regeneration is
required:

```bash
python eval/score.py --out-dir eval/outputs/aime/<TAG> --judge rule
```

When the LLM judge is used, `summary.json` additionally reports
`rule_vs_llm_agreement`, the fraction of rollouts on which both modes agree. On
AIME 2026 the two modes agreed on all 960 rollouts.

---

## 6. Outputs

Results are written to `eval/outputs/<benchmark>/<TAG>/`:

| File | Contents |
| :--- | :--- |
| `summary.json` | Aggregate metrics. `Pass@1` is the headline figure; `Pass@k`, `num_tasks`, `num_rollouts` and `coverage_complete` are also reported. |
| `scored.jsonl` | One row per rollout: raw model output, extracted answer, verdict. |
| `raw_rank*.jsonl` | Raw model outputs as written by each data-parallel rank. |
| `meta.json` | The complete configuration of the run, for reproducibility. |

A `Pass@1` figure is only valid when `coverage_complete` in `summary.json` is
`true`; otherwise one or more ranks or samples are missing.

---

## 7. Reference results

The following Pass@1 values are reference figures for the V4.1 model under the
default configuration described above.

| Benchmark | Pass@1 |
| :--- | ---: |
| AIME 2026 | 99.9% |
| GPQA Diamond | 93.4% |
| HLE-Full | 42.6% |

Due to the stochastic nature of LLM generation, repeated runs may differ by a small margin. 
These figures are therefore indicative rather than exact, and serve as a sanity check
that a deployment is configured correctly.