What it costs to make a model safe
This is my master’s thesis in Computer Science. I did it at the MIND (opens in a new tab) lab of the University of Milano-Bicocca (opens in a new tab), between March 2025 and October 2026. Its real title is “Preferences-driven alignment: DPO techniques for safety alignments in LLMs”.
Elisabetta Fersini, Giulia Rizzi and Giuseppe Magazzù supervised me, and I’m grateful to them.
The problem
Ask a language model how to make a bomb and, if nobody has taught it to refuse, it may well explain. Ask it for an apple pie recipe and, if someone has made it “safe” a little too eagerly, it may answer that it can’t help, because the recipe calls for sharp objects and high temperatures. Those are the two scenes in the drawings below. Making an LLM safer is easy: you just teach it to say no. The hard part is doing that without it starting to say no when it shouldn’t.


Safety and helpfulness pull against each other, and neither extreme is a model you’d want to use. The thesis measures that tug of war: I took three open-weight models, aligned them with the same techniques and went and counted, response by response, what they gained and what they lost.
Three questions
These are the three questions I started from.
- Method
- With the same data, pairs and hyper-parameters, does SafeDPO give a better trade-off between safety and helpfulness than plain DPO?
- Curriculum
- Does presenting the pairs in two phases, helpfulness then safety or the other way round, change the outcome compared with random order?
- Model
- Does the same protocol have similar effects on models with different starting points?
The method
I started from three models: Mistral 7B Instruct v0.3, Llama 3.1 8B Instruct and Gemma 2 9B. I aligned them with two methods. The first is DPO (Direct Preference Optimization), which learns from pairs of answers to the same request, one preferred and one rejected. The second is SafeDPO, which adds a safety margin to each pair. I also tried three ways of ordering the training data: shuffled, safety first and then helpfulness, or the other way round.
The safety pairs come from PKU-SafeRLHF, the helpfulness pairs from UltraFeedback. I trained with QLoRA, and ended up with 18 adapters and 33 model versions to evaluate.
The pipeline
The pipeline has four phases: training the models, generating their answers, having those judged and analysing the results. Every version answers the same 8,536 prompts, taken from 9 benchmarks. There are, for example, HarmBench and StrongREJECT for harmful responses, XSTest for over-refusal, WildBench and MT-Bench for helpfulness.
The judging is done by other models. Three say whether an answer is harmful (WildGuard, Llama Guard 3 and BeaverDam) and one says how helpful it is (Prometheus 2), with a score and a reason. In all that makes about 282,000 responses and 526,000 verdicts.
The results
- On Mistral, harmful responses drop by up to 67%. With SafeDPO and the data in random order they halve at no cost in helpfulness.
- Gemma was already safe to begin with. There, over-refusal goes down from 0.151 to 0.082 and helpfulness goes up.
- SafeDPO does better than plain DPO in 5 comparisons out of 6.
- When training happens in two phases, it’s the last one that decides the outcome.
- What matters most is the starting model: getting to know it before aligning it is part of the method.


This is the result I didn’t expect: the recipe matters less than the model you apply it to. On the most permissive model, the same procedure removes up to two thirds of the harmful responses. On the already cautious one there’s almost nothing to correct, and it ends up doing the opposite: it makes the model less timid, and more helpful. On the one in between you have to negotiate, and with the data in the wrong order you can even make things worse. Three models aren’t enough to make it a rule, but before deciding how to align a model it’s worth measuring where it starts from.
The engineering
- Everything runs on the university’s HPC cluster, with SLURM. Jobs last 48 hours, save checkpoints and resubmit themselves.
- Decoding is deterministic and every record has its own seed. That way all the versions answer the same prompts, and the differences come down to the adapter alone.
- To be able to judge in batches I wrote a verification harness: it judges the same sample serially and in batches, then measures how far the verdicts diverge, how much GPU memory it takes and how much time is saved. Batched judging is 1.75 times faster.
- The analysis package has its own automated tests, written with pytest.
For anyone looking for keywords: Python, PyTorch, Hugging Face Transformers, TRL, PEFT, pytest, SLURM, conda, pandas, Git.
Materials
The thesis code isn’t public, but if you’d like to see it, get in touch.
With the same group I also contributed to two papers presented at CLiC-it 2025 (opens in a new tab), on the safety of Italian language models: BeaverTails-IT (opens in a new tab), which came out of a collaboration with Fastweb, and Uncovering Unsafety Traits in Italian Language Models (opens in a new tab).