Preferences-driven alignment: DPO techniques for safety alignments in LLMs
This is my master’s thesis, done at the university’s MIND (opens in a new tab) laboratory. I compared three open-weight language models (Mistral, Llama and Gemma) before and after alignment with DPO and SafeDPO, two techniques that teach a model which of two answers to prefer, to see how much safer they become and how much helpfulness they lose: I evaluated 33 versions on 9 benchmarks, and harmful responses drop by up to 67%. I wrote the pipeline that trains them with QLoRA, queries them, has them judged by other models and analyses the results on the university HPC cluster, with automated tests and experiments reproducible line by line.



















