Methods like Doc-to-LoRA use a hypernetwork to turn a document into LoRA weights almost instantly. They work well until the document disagrees with what the model learned in pre-training. Then the old knowledge tends to win.
We studied why. The size of the adapter’s update and the strength of the prior it has to overcome turn out to be closely linked, and the failures concentrate where priors are strongest. We propose two fixes, Selective Layer Boosting and Conflict-Aware Internalization. On deep conflicts, accuracy rises from 46.4% to 71.0% on Gemma-2B and from 53.6% to 72.5% on Mistral-7B, while recall of genuinely new facts on Gemma stays at 97.1%. A new benchmark, KID-Bench, separates new-fact recall, composition, and conflicts of different prior strength.
The first spark came from somewhere else entirely: Deleuze’s syntheses of time and his dark precursor. The paper makes its own technical case; the story of how I got there is separate.