benthecarman
8ae8526c9d
* Cap ARA's neighbor count at the number of prompts The nearest neighbors are selected from the outputs for the good and bad prompts, which have one row per prompt. With fewer than 15 prompts in either dataset, the optimizer could pick a neighbor count larger than that, and torch.topk would crash with "selected index k out of range". Bounding the search range instead of clamping the value keeps the optimizer from wasting trials on counts that would all behave the same. * Render chat templates with a fixed date Some chat templates, including gpt-oss's, put the current date in the system prompt. That makes the residuals, and with them the modified model, depend on the day Heretic runs, so a model could not be reproduced on a different day. It also made the gpt-oss test's hash change daily once ARA started modifying the model. Prompts used for computing residuals and for evaluation are now rendered with a fixed date. Interactive chat keeps the real one. * Prevent ARA from sampling an empty layer range The start index was sampled from [0, L//2] and the exclusive end index from [L//2, L], so both could land on L//2 and leave no layers to modify, wasting the trial. Starting the end range at L//2 + 1 rules this out while keeping the bounds fixed, as multivariate TPE requires. On the 2-layer test models, every ARA trial had picked this empty range, so the seed-oss and gpt-oss tests never ran the optimization. They now modify layer 1, which changes their expected hashes. The seed-oss output also differs between the CPUs of GitHub's runners, because L-BFGS amplifies small floating-point differences, so it gets CI hashes as well.
Test Suite Guide
Whenever we change any code-logic related to src/heretic/model.py or config.toml (e.g. row_normalization, full_normalization_lora_rank, winsorization_quantile, etc) which can affect a model's reproduciblity; Use these tests which are designed to verify that those changes does not affect reproducibility, unless they are meant to (like when we'll integrate ARA branch in future).
How to test
- Choose any model from tiny-random org which provides tiny models useful for debugging.
Example: tiny-random/minicpm5.
Note
It is highly recommended to use a model which does not have a
special_tokens_map.jsonfile in the repo. Because those files are almost always wrong intiny-random/*models compared to the original model.
- Clone that model repository using Git and generate the SHA256 hashes using
sha256sum:
On Linux:
sha256sum -b * > ../SHA256SUMS.LABEL
On Windows:
sha256sum * | Out-File -Encoding utf8NoBOM ../SHA256SUMS.LABEL
Tip
On windows,
sha256sumis generally pre-installed by Git for windows.
Verify with:
Get-Command sha256sum`
Expected:
CommandType Name Version Source
----------- ---- ------- ------
Application sha256sum.exe 0.0.0.0 C:\Program Files\Git\usr\bin\sha256sum...
Note
You must use Windows Powershell
v7.Xnot the core which isv5.1. This is required for-Encoding utf8NoBOMto work.See Differences between Windows PowerShell 5.1 and PowerShell 7.x documentation.
Where LABEL describes the type of system you are running the tests on.
Example:
SHA256SUMS.windows(For windows)SHA256SUMS.ci(For GitHub CI)SHA256SUMS.linux(For linux)
- Run the tests with:
uv run run_tests.py
The output hashes should FAIL against the Valid hashes in SHA256SUMS file of the test model you added. This is expected since Heretic changes the model. Without Step 2, the test model's folder will simply be ignored because it will not have a hash SUMS file to compare against.
- After that go to the output
TEST_MODEL_DIR/modelfolder and re-generate the Actual hashes based on the system you are using.
cd TEST_MODEL_DIR/model
sha256sum -b * > ../SHA256SUMS.LABEL # or use windows command.
- Re-run the tests with:
uv run run_tests.py
This time the tests should PASS because we added the new hashes which are expected to be reproduced on the same system.
- After that push the
SHA256SUMS.LABELfiles and wait for GitHub CI actions to run those tests.
Since PyTorch does not guarantee exact cross-system reproducibility regardless of configuration, multiple valid hashes can be provided for each output file. The above update must be performed for each TEST_MODEL_DIR and on each type of system.
For this, copy the Actual hash value for each mismatched unidentical file into a SHA256SUMS.ci file.
- After that push the
SHA256SUMS.cifiles and wait for GitHub CI actions to re-run those tests.
This time the tests should PASS because we added the new hashes which are expected to be reproduced on CI.