fix: use W_org matrix only where needed (#398)

* fix: minor change use `W_org` matrix where needed... * Update model.py * Update model.py * fix: Windows hash, remove BOM marker * docs: Add info about test cases * feat: Tests for row_normalization PRE & NONE * feat: CI hash files for row_normalization PRE & NONE models * feat: Documentation instructions about test suite * add recommendation
2026-09-26 14:01:22 -07:00 · 2026-07-01 16:13:14 +05:30
parent 680c43e1bf
commit 7470dfd7af
14 changed files with 215 additions and 15 deletions
@@ -1,17 +1,90 @@
-Run the tests with
+# Test Suite Guide

-```sh
-uv run run_tests.py
-```
+Whenever we change any code-logic related to `src/heretic/model.py` or `config.toml` *(e.g. `row_normalization`, `full_normalization_lora_rank`, `winsorization_quantile`, etc)* which can affect a model's reproduciblity; Use these tests which are designed to verify that those changes does not affect reproducibility, unless they are meant to (like when we'll integrate ARA branch in future).

-To update the hashes after a logic change, run the tests, then execute
+## How to test

-```sh
-cd TEST_DIR/model
+1. Choose any model from [tiny-random](https://huggingface.co/tiny-random) org which provides tiny models useful for debugging. 
+
+**Example**: [tiny-random/minicpm5](https://huggingface.co/tiny-random/minicpm5).
+
+> [!NOTE]
+> It is highly recommended to use a model which does not have a `special_tokens_map.json` file in the repo.
+> Because those files are almost always wrong in `tiny-random/*` models compared to the original model.
+
+2. Clone that model repository using Git and generate the SHA256 hashes using `sha256sum`:
+
+**On Linux**:
+
+```bash
 sha256sum -b * > ../SHA256SUMS.LABEL
 ```

-where `LABEL` describes the type of system you are running the tests on.
-Since PyTorch does not guarantee exact cross-system reproducibility regardless of configuration,
-multiple valid hashes can be provided for each output file. The above update must be performed
-for each `TEST_DIR` and on each type of system.
+**On Windows**:
+
+```bash
+sha256sum * | Out-File -Encoding utf8NoBOM ../SHA256SUMS.LABEL
+```
+
+> [!TIP]
+> On windows, `sha256sum` is generally pre-installed by *Git for windows*.
+
+**Verify with**:
+
+```bash
+Get-Command sha256sum`
+```
+
+**Expected**:
+
+```bash
+CommandType     Name                                               Version    Source
+-----------     ----                                               -------    ------
+Application     sha256sum.exe                                      0.0.0.0    C:\Program Files\Git\usr\bin\sha256sum...
+```
+
+> [!NOTE]
+> You must use Windows Powershell `v7.X` not the core which is `v5.1`. This is required for `-Encoding utf8NoBOM` to work.
+>
+> See [Differences between Windows PowerShell 5.1 and PowerShell 7.x](https://learn.microsoft.com/en-us/powershell/scripting/whats-new/differences-from-windows-powershell?view=powershell-7.6) documentation.
+
+Where `LABEL` describes the type of system you are running the tests on.
+
+**Example**: 
+
+- `SHA256SUMS.windows` (For windows)
+- `SHA256SUMS.ci` (For GitHub CI)
+- `SHA256SUMS.linux` (For linux)
+
+3. Run the tests with:
+
+```bash
+uv run run_tests.py
+```
+
+The output hashes *should FAIL* against the `Valid hashes` in `SHA256SUMS` file of the test model you added. This is expected since Heretic changes the model. Without **Step 2**, the test model's folder will simply be ignored because it will not have a hash SUMS file to compare against.
+
+4. After that go to the output `TEST_MODEL_DIR/model` folder and re-generate the Actual hashes based on the system you are using.
+
+```bash
+cd TEST_MODEL_DIR/model
+sha256sum -b * > ../SHA256SUMS.LABEL # or use windows command.
+```
+
+5. Re-run the tests with:
+
+```bash
+uv run run_tests.py
+```
+
+This time the tests *should PASS* because we added the new hashes which are expected to be reproduced on the same system.
+
+6. After that push the `SHA256SUMS.LABEL` files and wait for GitHub CI actions to run those tests.
+
+Since PyTorch does not guarantee exact cross-system reproducibility regardless of configuration, multiple valid hashes can be provided for each output file. The above update must be performed for each `TEST_MODEL_DIR` and on each type of system.
+
+For this, copy the `Actual hash` value for *each mismatched unidentical* file into a `SHA256SUMS.ci` file.
+
+7. After that push the `SHA256SUMS.ci` files and wait for GitHub CI actions to re-run those tests.
+
+This time the tests *should* PASS because we added the new hashes which are expected to be reproduced on CI.