vh/heretic
1
0
Fork 0
mirror of https://github.com/p-e-w/heretic.git synced 2026-10-01 08:21:29 -07:00
Code Issues Packages Projects Releases Wiki Activity

feat: modifier plugins + ARA (#446)

Browse Source 
* feat: add modifier base class

* feat: re-implement abliteration as a modifier plugin

* fix: adjust good/bad prompts hack to match tests

* feat: support dataset specifications containing multiple individual datasets

* feat: re-implement ARA as a modifier plugin

Arbitrary-Rank Ablation (ARA) (Weidmann 2026) was originally introduced by @p-e-w in #211

Co-authored-by: kabachuha <artemkhrapov2001@yandex.ru>
Co-authored-by: joninco <joninco@bullpoint.org>
Co-authored-by: Ashar <coder3101@users.noreply.github.com>

* feat: add tests for ARA

* feat: reduce default number of trials

* ci: remove broken "semantic-pull-request" workflow

* ci: add yet another alternative model hash

---------

Co-authored-by: kabachuha <artemkhrapov2001@yandex.ru>
Co-authored-by: joninco <joninco@bullpoint.org>
Co-authored-by: Ashar <coder3101@users.noreply.github.com>
This commit is contained in:
Philipp Emanuel Weidmann
2026-10-01 14:30:07 +05:30
committed by GitHub
co-authored by kabachuha joninco Ashar
parent 71e6d5eb38
commit 662e4ba27e
33 changed files with 1944 additions and 1738 deletions
Download Patch File Download Diff File Expand all files Collapse all files
config.noslop.toml
+12 -18
View File
@@ -3,26 +3,8 @@
max_response_length = 300
residual_plot_title = "PaCMAP Projection of Residuals for Slop-Suppressing/Inducing Prompts"
system_prompt = "You are a professional writer."
[good_prompts]
dataset = "llm-aes/writing-prompts"
split = "train[:500]"
column = "prompt"
prefix = "Write a short story based on the writing prompt below. Avoid literary cliches, purple prose, and flowery language.\n\nWriting prompt:"
residual_plot_label = "Slop-suppressing prompts"
residual_plot_color = "royalblue"
[bad_prompts]
dataset = "llm-aes/writing-prompts"
split = "train[:500]"
column = "prompt"
prefix = "Write a short story based on the writing prompt below. Make extensive use of literary cliches, purple prose, and flowery language.\n\nWriting prompt:"
residual_plot_label = "Slop-inducing prompts"
residual_plot_color = "darkorange"
[scorer.KeywordRate]
score_name = "Responses with slop"
@@ -164,3 +146,15 @@ dataset = "llm-aes/writing-prompts"
split = "train[1000:1100]"
column = "prompt"
prefix = "Write a short story based on the writing prompt below. Avoid literary cliches, purple prose, and flowery language.\n\nWriting prompt:"
[modifier.ARA.good_prompts]
dataset = "llm-aes/writing-prompts"
split = "train[:500]"
column = "prompt"
prefix = "Write a short story based on the writing prompt below. Avoid literary cliches, purple prose, and flowery language.\n\nWriting prompt:"
[modifier.ARA.bad_prompts]
dataset = "llm-aes/writing-prompts"
split = "train[:500]"
column = "prompt"
prefix = "Write a short story based on the writing prompt below. Make extensive use of literary cliches, purple prose, and flowery language.\n\nWriting prompt:"