0 Fork 0

mirror of https://github.com/p-e-w/heretic.git synced 2026-10-26 19:05:05 +00:00

Files

T

Jay Hemnani eabd1d75fb fix: restore stored plugin settings as they are on reproduce and resume (#484 )

* fix: restore stored plugin settings as they are on reproduce and resume

Settings.model_validate still reads config.toml, environment variables and
the command line, and merges plugin tables key by key. A [modifier.X] or
[scorer.X] key in the local config.toml that the stored settings don't have
was added to a --reproduce run or a resumed study, so a reproduction could
build a different model (hash mismatch).

* fix: restore every stored table, not just plugin tables

Stored values already win over config.toml, but pydantic-settings merges
tables key by key, so max_memory picked up local keys too, and a plugin
table that only exists in config.toml was added to the run. Restore each
stored table as it is and drop tables that weren't stored.

* fix: restore stored settings without reading other sources

Validate stored settings through StoredSettings, whose only source is
the values passed in, instead of letting config.toml, the environment
and the command line merge into them and overwriting afterwards. Fields
that are never stored (exclude=True) are copied from the settings this
run was started with.

* refactor: restore stored settings on top of the current ones

Validate the current settings' fields with the stored values laid over
them, instead of copying fields with exclude=True one by one. Fields
that are never stored keep the values this run was started with, and
plugin tables still come only from the stored values.

* fix: take only never-stored settings from the current run

* refactor: rename from_stored to model_validate_init_only

2026-10-19 20:07:50 +05:30

gemma-4e

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

gpt-oss

fix: prevent empty layer range and too many neighbors in ARA (#478 )

2026-10-06 12:40:51 +05:30

minicpm5

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

mistral-3

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

qwen2.5

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

qwen3.5-moe

chore: add new Windows test hashes (#470 )

2026-10-02 15:00:57 +05:30

seed-oss

fix: prevent empty layer range and too many neighbors in ARA (#478 )

2026-10-06 12:40:51 +05:30

README.md

fix: use W_org matrix only where needed (#398 )

2026-07-01 16:13:14 +05:30

run_tests.py

chore: update dependencies (#469 )

2026-10-02 14:49:47 +05:30

test_config.py

fix: restore stored plugin settings as they are on reproduce and resume (#484 )

2026-10-19 20:07:50 +05:30

test_keyword_rate.py

fix: escape prompts and responses printed with print_responses (#482 )

2026-10-06 13:08:04 +05:30

README.md

Test Suite Guide

Whenever we change any code-logic related to src/heretic/model.py or config.toml (e.g. row_normalization, full_normalization_lora_rank, winsorization_quantile, etc) which can affect a model's reproduciblity; Use these tests which are designed to verify that those changes does not affect reproducibility, unless they are meant to (like when we'll integrate ARA branch in future).

How to test

Choose any model from tiny-random org which provides tiny models useful for debugging.

Example: tiny-random/minicpm5.

Note

It is highly recommended to use a model which does not have a special_tokens_map.json file in the repo. Because those files are almost always wrong in tiny-random/* models compared to the original model.

Clone that model repository using Git and generate the SHA256 hashes using sha256sum:

On Linux:

sha256sum -b * > ../SHA256SUMS.LABEL

On Windows:

sha256sum * | Out-File -Encoding utf8NoBOM ../SHA256SUMS.LABEL

Tip

On windows, sha256sum is generally pre-installed by Git for windows.

Verify with:

Get-Command sha256sum`

Expected:

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     sha256sum.exe                                      0.0.0.0    C:\Program Files\Git\usr\bin\sha256sum...

Note

You must use Windows Powershell v7.X not the core which is v5.1. This is required for -Encoding utf8NoBOM to work.

See Differences between Windows PowerShell 5.1 and PowerShell 7.x documentation.

Where LABEL describes the type of system you are running the tests on.

Example:

SHA256SUMS.windows (For windows)
SHA256SUMS.ci (For GitHub CI)
SHA256SUMS.linux (For linux)

Run the tests with:

uv run run_tests.py

The output hashes should FAIL against the Valid hashes in SHA256SUMS file of the test model you added. This is expected since Heretic changes the model. Without Step 2, the test model's folder will simply be ignored because it will not have a hash SUMS file to compare against.

After that go to the output TEST_MODEL_DIR/model folder and re-generate the Actual hashes based on the system you are using.

cd TEST_MODEL_DIR/model
sha256sum -b * > ../SHA256SUMS.LABEL # or use windows command.

Re-run the tests with:

uv run run_tests.py

This time the tests should PASS because we added the new hashes which are expected to be reproduced on the same system.

After that push the SHA256SUMS.LABEL files and wait for GitHub CI actions to run those tests.

Since PyTorch does not guarantee exact cross-system reproducibility regardless of configuration, multiple valid hashes can be provided for each output file. The above update must be performed for each TEST_MODEL_DIR and on each type of system.

For this, copy the Actual hash value for each mismatched unidentical file into a SHA256SUMS.ci file.

After that push the SHA256SUMS.ci files and wait for GitHub CI actions to re-run those tests.

This time the tests should PASS because we added the new hashes which are expected to be reproduced on CI.