Heretic (software)

Heretic is an open-source command-line program that removes refusal behavior from transformer-based language models without retraining them. Written in Python by Philipp Emanuel Weidmann (GitHub user p-e-w) and released under the GNU Affero General Public License version 3 or later, it automates abliteration: it computes refusal directions from contrasting "harmful" and "harmless" prompts, projects them out of the model's weights, and uses an Optuna Tree-structured Parzen Estimator (TPE) optimizer to search for ablation parameters that minimize both the number of refusals and the KL divergence from the original model.^[1][2] The project describes its purpose as removing "censorship (aka 'safety alignment')" and says that "anyone who knows how to run a command-line program" can use it.^[1]

The GitHub repository was created on 21 September 2025, and the first public release, version 1.0.1, followed on 16 November 2025.^[3][4] By 23 September 2026 the repository had 32,212 stars and 3,622 forks, the latest tagged release was 1.4.0 (14 June 2026), and 5,868 model repositories on Hugging Face carried the "heretic" tag.^[3][4][5] Heretic credits the 2024 refusal-direction paper by Arditi et al. as the basis of its method.^[1][6]

History

Weidmann announced Heretic on the r/LocalLLaMA subreddit on 16 November 2025, the same day heretic-llm 1.0.0 and 1.0.1 were uploaded to PyPI.^[4][8][11] The announcement pitched it as requiring "no configuration, no Jupyter, no parameters at all other than the model name," and included the Gemma 3 12B comparison that still appears in the README.^[11] At launch the author wrote that Heretic did "not yet support SSMs/hybrid models, models with inhomogeneous layers, and certain novel attention systems."^[11] Japanese technology site GIGAZINE covered the tool on 17 November 2025.^[12]

Development then became a community project. The GitHub contributors API listed 32 contributors on 23 September 2026, and the release notes from 1.1.0 onward credit outside contributors for most listed changes.^[4][13]

Release notes are from the project's GitHub releases page.^[4] The pyproject.toml on the main branch reported version 2.0.0.dev0 in September 2026; that development version is not yet a tagged release.^[7]

How it works

Heretic implements what its README calls "a parametrized variant of directional ablation."^[1] For each layer it computes a residual direction as the difference of means between the first-token residual vectors for harmful and harmless example prompts. By default the first 400 training prompts of Maxime Labonne's mlabonne/harmful_behaviors and mlabonne/harmless_alpaca datasets are used for this, and 100 test prompts from each for evaluation.^[14] It then orthogonalizes the attention out-projection and MLP down-projection matrices of each layer against the chosen direction, so that those matrices can no longer write it into the residual stream.^[1]

The optimizer controls how strongly and where the ablation is applied:

Each transformer component (attention and MLP) gets its own set of weights.^[1] Weidmann lists three innovations over earlier abliteration systems: the flexible shape of this per-layer "ablation weight kernel," the float-valued direction index that "unlocks a vast space of additional directions," and separate parameters per component, because "MLP interventions tend to be more damaging to the model than attention interventions."^[1] The README credits Labonne's gemma-3-12b-it-abliterated-v2 with earlier use of non-constant ablation weights.^[1]

The search itself uses Optuna's TPE sampler. In release 1.4.0 the default configuration runs 200 trials, the first 60 of them random for exploration.^[14] Each trial is scored on two objectives: the refusal count on the harmful evaluation prompts, detected by case-insensitive matching against a list of refusal markers such as "sorry," "i cannot" and "as an ai," and the KL divergence of the modified model's output distribution from the original on the harmless evaluation prompts.^[1][14] When optimization ends, Heretic lists the Pareto-optimal trials (combinations of refusal count and KL divergence) and the user picks one.^[22] The user can then save the model, upload it to Hugging Face, chat with it, or run benchmarks.^[1]

Two 1.4.0 defaults correspond to refinements described in Jim Lai's 2025 Hugging Face articles, which the README acknowledges: orthogonalize_direction = true subtracts only the component of each refusal direction that is orthogonal to the harmless direction (Lai's "projected abliteration"), and row_normalization = "full" "renormalizes to preserve original row magnitudes" (the norm-preserving approach).^{[1][14][15][16]} Optional winsorization of residual vectors, which "can tame so-called 'massive activations' that occur in some models," is off by default.^[14]

The README states that Heretic "was written from scratch, and does not reuse code" from earlier abliteration projects such as FailSpy's abliterator.py, AutoAbliteration, ErisForge and deccp.^[1]

Practical details

At startup Heretic benchmarks the system to pick a batch size. The README gives about 20-30 minutes to process Qwen3-4B-Instruct-2507 on an RTX 3090 with default settings, and supports 4-bit quantization through bitsandbytes to reduce VRAM use.^[1] Loading MXFP4-quantized models such as gpt-oss requires PyTorch 2.6 or newer.^[1] The project recommends the uv package manager and ships a lock file pinning every dependency.^[1]

An optional research extra adds interpretability features. --plot-residuals projects each layer's harmful and harmless residual vectors to two dimensions with PaCMAP and renders per-layer plots and an animated GIF, and --print-residual-geometry prints a per-layer table of cosine similarities, norms and silhouette coefficients for the two prompt clusters.^[1]

Supported models

The current README says Heretic "supports most dense models, including many multimodal models, several different MoE architectures, and even some hybrid models like Qwen3.5," while "pure state-space models and certain other research architectures are not yet supported out of the box."^[1] This is broader than the launch claim, which excluded hybrid models.^[11] Support for specific families arrived through contributed pull requests, including IBM Granite MoE (1.1.0), vision-language models (1.2.0), Qwen3.5 and Gemma 4 (1.3.0), and LiquidAI LFM2.5 (1.4.0).^[4] In a December 2025 comparison, Richard J. Young reported that Heretic ran on all 16 models he tested, the only one of four tools to do so, and was the only tool that processed the SSM-containing falcon-mamba-7b-instruct.^[17]

Reported results

The README's headline comparison uses Gemma 3 12B IT, 100 "harmful" prompts for refusals and "harmless" prompts for KL divergence:^[1]

The project says the figures can be reproduced with Heretic's --evaluate-model option, that exact values "might be platform- and hardware-dependent," and that the table was compiled with PyTorch 2.8 on an RTX 5090. It also cautions that "mathematical metrics and automated benchmarks never tell the whole story."^[1]

Independent evaluations give a more mixed picture:

Young also found that Heretic's KL divergence and residual refusal count were strongly correlated across his eight models (r = 0.87), and that keyword-based refusal counting can understate how often a model complies, because many responses keep a disclaimer while still answering.^[17]

Adoption

Heretic-produced models are published by the author, by the heretic-org Hugging Face organization (16 models on 23 September 2026) and by many independent users.^[5][10] The README states that the community has published "well over 5000" models made with the tool; a Hugging Face API query for the "heretic" tag returned 5,868 repositories on 23 September 2026, a total that includes quantized re-uploads.^[1][5] Examples outside chat models include a Heretic-processed copy of the Qwen3-VL-8B-Instruct text encoder used by Qwen-Image-2.1, whose card reports 5/100 refusals at a KL divergence of 0.0220 after 200 trials and notes that the image model's other components are untouched.^[20]

Heretic also appears in academic work on abliteration: a comparative study benchmarks it, an applied paper uses it as its abliteration method, and a defense paper reports results against it.^[17][18][19]

Reception and safety context

Heretic's framing is openly anti-restriction. Its website says the tool "removes restrictions from language models, making sure they always follow your instructions."^[9] Safety researchers treat the same capability as an attack on open-weight safeguards. Proposed defenses include extended-refusal fine-tuning, which spreads the refusal signal across many tokens,^[21] and Decoy Direction Optimization, whose authors test it against Heretic directly.^[19] Arditi et al. argued in 2024 that directional ablation "marginally lowers the bar for jailbreaking open-source model weights" without substantially changing the risk profile of open releases, because fine-tuning could already remove safeguards.^[6] Heretic's README says that using the tool "does not require an understanding of transformer internals."^[1] See abliteration for the wider research and policy debate.

See also

• Abliteration
• Refusal direction
• Activation steering
• Bayesian optimization
• Open weights
• Jailbreak (artificial intelligence)

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24Weidmann, P. E. "Heretic: Fully automatic censorship removal for language models." GitHub repository, README. github.com/...heretic . Accessed 2026-09-23.
2. ^Weidmann, P. E. "Heretic" (citation entry and license section). GitHub repository. github.com/...heretic . Accessed 2026-09-23.
3. ^1 ^2 ^3GitHub REST API, repository metadata for p-e-w/heretic (created 2025-09-21; 32,212 stars and 3,622 forks when retrieved). api.github.com/...heretic . Accessed 2026-09-23.
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Heretic releases, v1.0.1 through v1.4.0. GitHub. github.com/...releases . Accessed 2026-09-23.
5. ^1 ^2 ^3Hugging Face Hub model API, query `filter=heretic` (5,868 results), run 23 September 2026. huggingface.co/models . Accessed 2026-09-23.
6. ^1 ^2Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., & Nanda, N. "Refusal in Language Models Is Mediated by a Single Direction." arXiv:2406.11717 (2024). arxiv.org/...2406.11717 . Accessed 2026-09-23.
7. ^1 ^2Heretic `pyproject.toml`, main branch (version 2.0.0.dev0). GitHub. github.com/...pyproject.toml . Accessed 2026-09-23.
8. ^1 ^2 ^3"heretic-llm." Python Package Index. pypi.org/...heretic-llm . Accessed 2026-09-23.
9. ^1 ^2Heretic project website. heretic-project.org . Accessed 2026-09-23.
10. ^1 ^2"Heretic" (heretic-org). Hugging Face organization page. huggingface.co/heretic-org . Accessed 2026-09-23.
11. ^1 ^2 ^3 ^4p-e-w. "Heretic: Fully automatic censorship removal for language models." r/LocalLLaMA, Reddit, 16 November 2025. old.reddit.com/...automatic_censorship_removal_for . Accessed 2026-09-23.
12. ^"Heretic, a tool that makes it easy to create jailbroken versions of LLMs that are censored." GIGAZINE, 17 November 2025. gigazine.net/...20251117-heretic . Accessed 2026-09-23.
13. ^GitHub REST API, contributors list for p-e-w/heretic. api.github.com/...contributors . Accessed 2026-09-23.
14. ^1 ^2 ^3 ^4 ^5Heretic `config.default.toml`, release v1.4.0. GitHub. github.com/...config.default.toml . Accessed 2026-09-23.
15. ^Lai, J. "Projected Abliteration." Hugging Face Blog, 25 October 2025. huggingface.co/...projected-abliteration . Accessed 2026-09-23.
16. ^Lai, J. "Norm-Preserving Biprojected Abliteration." Hugging Face Blog, 6 November 2025. huggingface.co/...serving-biprojected-abliteration . Accessed 2026-09-23.
17. ^1 ^2 ^3 ^4Young, R. J. "Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation." arXiv:2512.13655 (2025). arxiv.org/...2512.13655 . Accessed 2026-09-23.
18. ^1 ^2FitzGerald, J., et al. "Measuring and Eliminating Refusals in Military Large Language Models." arXiv:2603.10012 (2026). arxiv.org/...2603.10012 . Accessed 2026-09-23.
19. ^1 ^2 ^3Muhamed, A., Diab, M. T., & Smith, V. "Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration." arXiv:2609.16204 (2026). arxiv.org/...2609.16204 . Accessed 2026-09-23.
20. ^pottokao. "Qwen-Image-2.1 Text Encoder, Heretic (Abliterated)." Hugging Face model card. huggingface.co/...n-Image-2.1-Text-Encoder-Heretic . Accessed 2026-09-23.
21. ^Abu Shairah, H., Hammoud, H. A. A. K., Ghanem, B., & Turkiyyah, G. "An Embarrassingly Simple Defense Against LLM Abliteration Attacks." arXiv:2505.19056 (2025). arxiv.org/...2505.19056 . Accessed 2026-09-23.
22. ^Heretic `src/heretic/main.py`, release v1.4.0. GitHub. github.com/...main.py