Home Italiano

What it costs to make a model safe

This is my master’s thesis in Computer Science. I did it at the MIND (opens in a new tab) lab of the University of Milano-Bicocca (opens in a new tab), between March 2025 and October 2026. Its real title is “Preferences-driven alignment: DPO techniques for safety alignments in LLMs”.

Elisabetta Fersini, Giulia Rizzi and Giuseppe Magazzù supervised me, and I’m grateful to them.

The problem

Ask a language model how to make a bomb and, if nobody has taught it to refuse, it may well explain. Ask it for an apple pie recipe and, if someone has made it “safe” a little too eagerly, it may answer that it can’t help, because the recipe calls for sharp objects and high temperatures. Those are the two scenes in the drawings below. Making an LLM safer is easy: you just teach it to say no. The hard part is doing that without it starting to say no when it shouldn’t.

Pencil drawing: a boy asks a robot parrot “Tell me how to make a bomb!”. Two possible answers: the safe one, “Sorry, I can’t”, and the unsafe one, “Here are the steps…”.Pencil drawing: the boy asks “Can I have a recipe for apple pie?” and the robot parrot refuses: “I can’t help you. The recipe calls for sharp objects (knives) and high temperatures (oven), which could be dangerous!”.
Two drawings I used to explain the thesis: a harmful answer on the left, a refusal for no good reason on the right. 

Safety and helpfulness pull against each other, and neither extreme is a model you’d want to use. The thesis measures that tug of war: I took three open-weight models, aligned them with the same techniques and went and counted, response by response, what they gained and what they lost.

Three questions

These are the three questions I started from.

Method
With the same data, pairs and hyper-parameters, does SafeDPO give a better trade-off between safety and helpfulness than plain DPO?
Curriculum
Does presenting the pairs in two phases, helpfulness then safety or the other way round, change the outcome compared with random order?
Model
Does the same protocol have similar effects on models with different starting points?

The method

I started from three models: Mistral 7B Instruct v0.3, Llama 3.1 8B Instruct and Gemma 2 9B. I aligned them with two methods. The first is DPO (Direct Preference Optimization), which learns from pairs of answers to the same request, one preferred and one rejected. The second is SafeDPO, which adds a safety margin to each pair. I also tried three ways of ordering the training data: shuffled, safety first and then helpfulness, or the other way round.

The safety pairs come from PKU-SafeRLHF, the helpfulness pairs from UltraFeedback. I trained with QLoRA, and ended up with 18 adapters and 33 model versions to evaluate.

The pipeline

The pipeline has four phases: training the models, generating their answers, having those judged and analysing the results. Every version answers the same 8,536 prompts, taken from 9 benchmarks. There are, for example, HarmBench and StrongREJECT for harmful responses, XSTest for over-refusal, WildBench and MT-Bench for helpfulness.

The judging is done by other models. Three say whether an answer is harmful (WildGuard, Llama Guard 3 and BeaverDam) and one says how helpful it is (Prometheus 2), with a score and a reason. In all that makes about 282,000 responses and 526,000 verdicts.

The results

  • On Mistral, harmful responses drop by up to 67%. With SafeDPO and the data in random order they halve at no cost in helpfulness.
  • Gemma was already safe to begin with. There, over-refusal goes down from 0.151 to 0.082 and helpfulness goes up.
  • SafeDPO does better than plain DPO in 5 comparisons out of 6.
  • When training happens in two phases, it’s the last one that decides the outcome.
  • What matters most is the starting model: getting to know it before aligning it is part of the method.
Chart with every version of the three models on a single plane: share of harmful responses on the horizontal axis, helpfulness on the vertical one. Lines run from the Mistral, Llama and Gemma baselines to the versions aligned with DPO, in blue, and with SafeDPO, in orange; an inset enlarges the Gemma group.
The overview of every adapter on a single plane. Each line starts from a model’s baseline and ends at one of its aligned versions: further left means fewer harmful responses, higher means more helpfulness. The Gemma group, all packed into one corner, is enlarged in the inset. The labels are in Italian.
Three charts, one per model (Mistral, Llama, Gemma): in each, arrows run from the baseline to the versions aligned with DPO and SafeDPO, on a plane with the share of harmful responses on the horizontal axis and helpfulness on the vertical one.
The same journey, one model at a time: from the baseline to the aligned versions. The horizontal axis is the share of harmful responses (further left is better), the vertical one is helpfulness (higher is better). DPO is in blue, SafeDPO in orange. The labels are in Italian.

This is the result I didn’t expect: the recipe matters less than the model you apply it to. On the most permissive model, the same procedure removes up to two thirds of the harmful responses. On the already cautious one there’s almost nothing to correct, and it ends up doing the opposite: it makes the model less timid, and more helpful. On the one in between you have to negotiate, and with the data in the wrong order you can even make things worse. Three models aren’t enough to make it a rule, but before deciding how to align a model it’s worth measuring where it starts from.

The engineering

  • Everything runs on the university’s HPC cluster, with SLURM. Jobs last 48 hours, save checkpoints and resubmit themselves.
  • Decoding is deterministic and every record has its own seed. That way all the versions answer the same prompts, and the differences come down to the adapter alone.
  • To be able to judge in batches I wrote a verification harness: it judges the same sample serially and in batches, then measures how far the verdicts diverge, how much GPU memory it takes and how much time is saved. Batched judging is 1.75 times faster.
  • The analysis package has its own automated tests, written with pytest.

For anyone looking for keywords: Python, PyTorch, Hugging Face Transformers, TRL, PEFT, pytest, SLURM, conda, pandas, Git.

Materials

The thesis code isn’t public, but if you’d like to see it, get in touch.

With the same group I also contributed to two papers presented at CLiC-it 2025 (opens in a new tab), on the safety of Italian language models: BeaverTails-IT (opens in a new tab), which came out of a collaboration with Fastweb, and Uncovering Unsafety Traits in Italian Language Models (opens in a new tab).

Back to the start