0 Fork 0

mirror of https://github.com/p-e-w/heretic.git synced 2026-10-26 19:05:05 +00:00

Files

T

History

Vinay Umretheandbenthecarman d34dd9192d fix: Improve response prefix logic, handle default behaviors, and clarify documentation (#523 )

* fix: Handle the default thinking behavior

Some chat-templates insert full CoT block when thinking is *disabled* by Default like Qwen3.5-2B. And some models by Default only insert the CoT initializer (somtimes possibly with whitespaces) like Qwen3.5-4B when thinking is *enabled*.

Co-authored-by: benthecarman <benthecarman@live.com>

* chore: Update CI test hashes

Removing $ from pattern_initializer broke the test model mistral-3, which was expected

* fix: Several fixes, docs update, cleanup

* chore: Update Windows test hash

very much expected, and looks good to me

* chore: Update CI test hashes

* chore: Update CI test hash (alternate)

good to see that it caught the third hash as well. Anyway, the GitHub CI remains terrible for such tests

* fix: minor change

* fix: Override the system prompt for the dummy_prompt case

This is correct for models which insert instructions about 'How to think' in the system prompt, which can cause false-positives when searching the dummy_prompt

* chore: Update CI test hashes

* chore: Update CI test hashes

* fix: minor change

* fix: Add alternate CI test hash

I'm not liking having a fourth hash btw

---------

Co-authored-by: benthecarman <benthecarman@live.com>

2026-10-26 23:43:56 +05:30

deepseek-v4

fix: Improve response prefix logic, handle default behaviors, and clarify documentation (#523 )

2026-10-26 23:43:56 +05:30

deepseek-v4-ara

fix: Improve response prefix logic, handle default behaviors, and clarify documentation (#523 )

2026-10-26 23:43:56 +05:30

gemma-4e

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

gpt-oss

fix: prevent empty layer range and too many neighbors in ARA (#478 )

2026-10-06 12:40:51 +05:30

minicpm5

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

mistral-3

fix: Improve response prefix logic, handle default behaviors, and clarify documentation (#523 )

2026-10-26 23:43:56 +05:30

qwen2.5

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

qwen3.5-moe

fix: Improve response prefix logic, handle default behaviors, and clarify documentation (#523 )

2026-10-26 23:43:56 +05:30

seed-oss

fix: prevent empty layer range and too many neighbors in ARA (#478 )

2026-10-06 12:40:51 +05:30

README.md

fix: use W_org matrix only where needed (#398 )

2026-07-01 16:13:14 +05:30

run_tests.py

chore: update dependencies (#469 )

2026-10-02 14:49:47 +05:30

test_chat_template.py

fix: support models whose chat template rejects the system role (#498 )

2026-10-20 21:47:51 +05:30

test_config.py

feat: make upload tags configurable (#506 )

2026-10-20 22:12:06 +05:30

test_keyword_rate.py

fix: reject blank refusal keyword markers (#512 )

2026-10-23 18:55:58 +05:30

README.md

Test Suite Guide

Whenever we change any code-logic related to src/heretic/model.py or config.toml (e.g. row_normalization, full_normalization_lora_rank, winsorization_quantile, etc) which can affect a model's reproduciblity; Use these tests which are designed to verify that those changes does not affect reproducibility, unless they are meant to (like when we'll integrate ARA branch in future).

How to test

Choose any model from tiny-random org which provides tiny models useful for debugging.

Example: tiny-random/minicpm5.

Note

It is highly recommended to use a model which does not have a special_tokens_map.json file in the repo. Because those files are almost always wrong in tiny-random/* models compared to the original model.

Clone that model repository using Git and generate the SHA256 hashes using sha256sum:

On Linux:

sha256sum -b * > ../SHA256SUMS.LABEL

On Windows:

sha256sum * | Out-File -Encoding utf8NoBOM ../SHA256SUMS.LABEL

Tip

On windows, sha256sum is generally pre-installed by Git for windows.

Verify with:

Get-Command sha256sum`

Expected:

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     sha256sum.exe                                      0.0.0.0    C:\Program Files\Git\usr\bin\sha256sum...

Note

You must use Windows Powershell v7.X not the core which is v5.1. This is required for -Encoding utf8NoBOM to work.

See Differences between Windows PowerShell 5.1 and PowerShell 7.x documentation.

Where LABEL describes the type of system you are running the tests on.

Example:

SHA256SUMS.windows (For windows)
SHA256SUMS.ci (For GitHub CI)
SHA256SUMS.linux (For linux)

Run the tests with:

uv run run_tests.py

The output hashes should FAIL against the Valid hashes in SHA256SUMS file of the test model you added. This is expected since Heretic changes the model. Without Step 2, the test model's folder will simply be ignored because it will not have a hash SUMS file to compare against.

After that go to the output TEST_MODEL_DIR/model folder and re-generate the Actual hashes based on the system you are using.

cd TEST_MODEL_DIR/model
sha256sum -b * > ../SHA256SUMS.LABEL # or use windows command.

Re-run the tests with:

uv run run_tests.py

This time the tests should PASS because we added the new hashes which are expected to be reproduced on the same system.

After that push the SHA256SUMS.LABEL files and wait for GitHub CI actions to run those tests.

Since PyTorch does not guarantee exact cross-system reproducibility regardless of configuration, multiple valid hashes can be provided for each output file. The above update must be performed for each TEST_MODEL_DIR and on each type of system.

For this, copy the Actual hash value for each mismatched unidentical file into a SHA256SUMS.ci file.

After that push the SHA256SUMS.ci files and wait for GitHub CI actions to re-run those tests.

This time the tests should PASS because we added the new hashes which are expected to be reproduced on CI.