mirror of
https://github.com/p-e-w/heretic.git
synced 2026-10-26 19:05:05 +00:00
benthecarman
80f81c90be
DeepSeek V4 and GLM 5.3 have several residual streams per token instead of one, so their hidden states have an extra dimension. Heretic crashed computing refusal directions for GLM 5.3, and found no modules to abliterate in DeepSeek V4. Use the mean of the streams as the residual. The streams are mixed with each other by a doubly stochastic matrix, so their mean only changes by what the blocks add to it, like an ordinary residual. Also abliterate DeepSeek V4's attention output (o_b_proj) and the shared experts of MoE models. Both models fuse their routed experts into one module, so the shared experts are the only MLP weights Heretic can edit.
2 lines
92 B
Plaintext
2 lines
92 B
Plaintext
9de2b756fa643074ac3641d97076d39c9f89e08a24a39a546642a59d975442f5 *adapter_model.safetensors
|