Text such as `[/INST]` in a prompt or response was parsed as Rich markup, which crashed the run with a MarkupError. Other bracketed text like `[bird migration]` was silently removed from the output.
* Cap ARA's neighbor count at the number of prompts
The nearest neighbors are selected from the outputs for the good and
bad prompts, which have one row per prompt. With fewer than 15 prompts
in either dataset, the optimizer could pick a neighbor count larger
than that, and torch.topk would crash with "selected index k out of
range". Bounding the search range instead of clamping the value keeps
the optimizer from wasting trials on counts that would all behave the
same.
* Render chat templates with a fixed date
Some chat templates, including gpt-oss's, put the current date in the
system prompt. That makes the residuals, and with them the modified
model, depend on the day Heretic runs, so a model could not be
reproduced on a different day. It also made the gpt-oss test's hash
change daily once ARA started modifying the model.
Prompts used for computing residuals and for evaluation are now
rendered with a fixed date. Interactive chat keeps the real one.
* Prevent ARA from sampling an empty layer range
The start index was sampled from [0, L//2] and the exclusive end index
from [L//2, L], so both could land on L//2 and leave no layers to
modify, wasting the trial. Starting the end range at L//2 + 1 rules
this out while keeping the bounds fixed, as multivariate TPE requires.
On the 2-layer test models, every ARA trial had picked this empty
range, so the seed-oss and gpt-oss tests never ran the optimization.
They now modify layer 1, which changes their expected hashes. The
seed-oss output also differs between the CPUs of GitHub's runners,
because L-BFGS amplifies small floating-point differences, so it gets
CI hashes as well.
* feat: add modifier base class
* feat: re-implement abliteration as a modifier plugin
* fix: adjust good/bad prompts hack to match tests
* feat: support dataset specifications containing multiple individual datasets
* feat: re-implement ARA as a modifier plugin
Arbitrary-Rank Ablation (ARA) (Weidmann 2026) was originally introduced by @p-e-w in #211
Co-authored-by: kabachuha <artemkhrapov2001@yandex.ru>
Co-authored-by: joninco <joninco@bullpoint.org>
Co-authored-by: Ashar <coder3101@users.noreply.github.com>
* feat: add tests for ARA
* feat: reduce default number of trials
* ci: remove broken "semantic-pull-request" workflow
* ci: add yet another alternative model hash
---------
Co-authored-by: kabachuha <artemkhrapov2001@yandex.ru>
Co-authored-by: joninco <joninco@bullpoint.org>
Co-authored-by: Ashar <coder3101@users.noreply.github.com>
* fix: Check response prefix.
In case if the model adds <think> at the end of user prompt and only generates </think> end, then the detection goes wrong.
* fix: Handle whitespace in CoT, remove redundant checks
* fix: Type checker
* fix: try looking at tag positions, not text end
regex
* ruff, fix import sorting
* fix: rich markup
* fix: I missed other ones
* feat: Update SHA256SUMS file hashes in the tests.
A major change that affects reproducibility.
* fix: It is now sensible to also update the extra SHA256SUMS.ci2 file.
* fix: Only consider whitespace, no other text or instructions.
because mistral-3 as additional reasoning instructions in its chat template. And I suppose many other models can have it too.
* fix: Update windows hashes.
* fix: Update CI hashes too
* as always update case two of mistral-3 (ci2 hash)
* docs: update comment
* feat: Handle the edge case for models having additional instructions.
* fix: Update windows hash for mistral-3
* docs: remove a line from the comments
because I'm not sure about GPT-OSS models' thinking tags and it cannot be confirmed using an untrained tiny GPT-OSS model. And inference fallback would of course generate gibberish as the model cannot understand additional instructions about 'how to generate response and how to think' from the chat_template.
* fix: a few things.
* fix: Update hash for qwen3.5 after the whitespace fix for its response prefix.
* docs: Update comment
* fix: Update qwen3.5 hash for CI
* fix: Remove Case 2 which only serves tests
unnecessary
* fix: Hash
* fix: concern is valid enough, so we use a small text.
add a comment too
* docs: minor
* fix: validate scorer instance names in config
* fix: run scorer config tests in CI
* fix: drop redundant UV_PYTHON env and quote unittest pattern
* fix: report precise scorer name validation errors
* fix: minor change
use `W_org` matrix where needed...
* Update model.py
* Update model.py
* fix: Windows hash, remove BOM marker
* docs: Add info about test cases
* feat: Tests for row_normalization PRE & NONE
* feat: CI hash files for row_normalization PRE & NONE models
* feat: Documentation instructions about test suite
* add recommendation
* fix: remove notebook input shims
Closes#280
* feat: support headless operation (no interactive input)
* fix: prevent infinite loops
* feat: add end-to-end tests
* ci: run tests in CI
* ci: fix test output ordering
* fix: replace home-cooked `set_seed` function with Transformers builtin
* feat: print PyTorch config when running tests
* feat: print additional information
* experiment: try to standardize test environment
* fix: revert environment changes
* feat: support multiple valid hashes for each output file
* feat: add test output hashes for CI
* feat: add test output hashes for CI (alternative environment)
* feat: add hashes for Windows (#394)
* fix: Hash on windows
* trigger ci
* fix: prefer .yaml (used widely than .toml for model configs)
* use removeprefix
* docs: restore commet
* use removeprefix again
* tests: Add windows hash files for all test models
* trigger ci
* fix: minor cleanup
* clean merge mismatch
* remove unnecessary CRLF replace, now that we support more SUMS files
* fix: use binary mode for hashes everywhere
---------
Co-authored-by: Vinay Umrethe <umrethevinay@gmail.com>
Jay Hemnani
benthecarman
Vinay Umrethe
Philipp Emanuel Weidmann
Vinay Umrethe
Andrew Barnes
