KAIST researchers have developed a training framework that stops large language models from losing their safety guardrails when fine-tuned on personal or corporate data. The method, called Buffer-and-Reinforce, was presented at the International Conference on Machine Learning (ICML) 2026 and allows customized models to perform new tasks while becoming even safer than the original.
The technology addresses a growing problem as organizations race to build personalized AI by retraining models like ChatGPT on their own documents. Fine-tuning improves task-specific performance, but it often weakens the safety mechanisms that prevent harmful outputs. A team led by Professor Changick Kim from KAIST's School of Electrical Engineering, with doctoral student Seokil Ham as first author, designed a two-stage process that separates learning from safety reinforcement.
The fine-tuning safety trade-off
Prior research had shown a counterintuitive effect: fine-tuning a model while it is in a temporarily jailbroken state-where it might answer dangerous requests-does not significantly degrade its safety. The KAIST team investigated why this happens and found that in the jailbroken state, the model becomes less susceptible to harmful information while still absorbing new task capabilities. Essentially, the model can learn useful knowledge without picking up harmful behaviors.
Based on this insight, they built a buffering module called BufferLoRA that acts as a temporary protective layer during fine-tuning. Once training on user data is complete, this module is removed, so the jailbroken state is never exposed in actual service.
Buffer-and-Reinforce: A two-stage approach
The framework has two phases. First, the buffering stage applies BufferLoRA to shield the base model from harmful data while allowing it to learn the desired tasks. After fine-tuning, the module is discarded.
Then, a safety reinforcement module, ReinforceLoRA, is applied to restore and strengthen the model's safety. The team used QR decomposition, a mathematical technique that separates different types of information, to selectively reinforce safety without erasing the new functions the user wanted. This means the model retains its customized performance while its safety safeguards are improved beyond the original state.
Experimental results
In tests where all user data consisted of harmful question-answer pairs, the fine-tuned model generated harmful responses only 8% of the time, compared to roughly 18% for the original, unmodified model. The framework achieved state-of-the-art safety without compromising on task performance, even under adversarial conditions.
"This research provides a key foundational technology that allows anyone to build customized AI with their own data while using it more safely," Kim said. "We expect it to contribute significantly to building a trustworthy AI service environment in the era of personalized AI and AI agents." The paper, authored by Seokil Ham et al., is available on arXiv and was selected for a Spotlight presentation at ICML 2026.
Why this matters for IT & Development
For IT teams and developers building custom AI tools, the Buffer-and-Reinforce framework removes a major barrier to safe deployment. Fine-tuning Generative AI and LLM models on internal documentation, codebases, or customer data can now be done without inadvertently creating a model that complies with dangerous requests. The method works without extra safety datasets and keeps computational costs low, making it practical for real-world enterprise use. As organizations move toward personalized AI for IT & Development tasks, this research offers a direct path to safer, more trustworthy systems.
Your membership also unlocks:
