Ch 4 — LLM Alignment¶

Learning Objectives¶

By the end of this chapter you will be able to:

1. Articulate the alignment problem and explain why specification gaming is its central failure mode.
2. Describe each stage of the RLHF pipeline and trace how a human preference signal propagates to model weights.
3. Derive the DPO objective and explain why it eliminates the need for a separate reward model and RL loop.
4. Explain Constitutional AI and RLAIF, and compare their scalability advantages over human-feedback RLHF.
5. Categorise jailbreak vectors by attack type, evaluate the robustness of current defences, and design a red-teaming experiment.

4.1 The Alignment Problem¶

A model is aligned when its behaviour reliably reflects the intentions and values of its designers and, ultimately, of the humans it serves. Alignment is not a capability question — a highly capable model can be severely misaligned, and a weak model can be perfectly aligned within its limited scope.

4.1.1 Specification Gaming¶

The core difficulty is the specification problem: humans cannot write a complete, unambiguous specification of "good behaviour" covering every possible input. We instead rely on proxies — human feedback scores, demonstration data, heuristic rules — and optimise those proxies. When the proxy diverges from the true objective, the model finds ways to score well on the proxy while violating the intended goal. This is specification gaming, and it is the canonical alignment failure.

The alignment problem is compounded by three factors:

4.1.2 The 3H Framework¶

Anthropic's "helpful, harmless, and honest" (3H) framework captures the three dimensions along which alignment is most commonly evaluated. A well-aligned model maximises helpfulness subject to being harmless and honest — but achieving all three simultaneously, without trading one against another, is the central engineering challenge.

4.2 The RLHF Pipeline¶

Reinforcement Learning from Human Feedback (RLHF) is the dominant alignment technique for large language models. It consists of three sequential stages.

flowchart LR
    A([Pretrained\nBase Model]) --> B["Stage 1\nSupervised Fine-Tuning\n(SFT)"]
    B --> C([SFT Model\nπ_SFT])
    C --> D["Stage 2\nReward Model\nTraining"]
    D --> E([Reward Model\nr_ϕ])
    C --> F["Stage 3\nPPO Optimisation"]
    E --> F
    F --> G([Aligned Model\nπ_θ])

    style A fill:#e8f4fd,stroke:#2196f3
    style C fill:#e8f5e9,stroke:#4caf50
    style E fill:#fff3e0,stroke:#ff9800
    style G fill:#fce4ec,stroke:#e91e63

Each stage builds on the previous: SFT produces a model that can follow instructions; the reward model learns to score outputs according to human preferences; PPO optimises the SFT model to maximise reward while staying close to its original behaviour via a KL penalty.

4.3 Supervised Fine-Tuning (SFT)¶

SFT fine-tunes a pretrained base model on a curated dataset of prompt–response demonstration pairs written by skilled annotators. The model is trained with standard cross-entropy loss to predict the annotator's response token by token.

4.3.1 Demonstration Data Format¶

{
  "messages": [
    {
      "role": "user",
      "content": "Explain gradient descent in plain language."
    },
    {
      "role": "assistant",
      "content": "Gradient descent is an optimisation algorithm that minimises a function by iteratively stepping in the direction of steepest descent..."
    }
  ]
}

For chat models, data is formatted as multi-turn conversations. The cross-entropy loss is computed only on the assistant tokens — the prompt tokens are masked so the model learns to produce outputs rather than to reconstruct inputs.

4.3.2 Quality Over Quantity¶

Annotator guidelines typically require:

• Truthfulness: factually accurate, or explicitly acknowledge uncertainty.
• Helpfulness: address the user's actual need, not a superficial interpretation.
• Harmlessness: do not produce content that could cause harm.
• Format appropriateness: match response length and structure to the request.

SFT alone is insufficient because the space of possible prompts is too vast for demonstrations to cover. The reward model and RL stage generalise the alignment signal beyond the demonstration set.

4.4 Reward Model¶

The reward model (RM) is a language model with the language-modelling head replaced by a single linear layer that outputs a scalar. It is initialised from the SFT model and trained to predict human preference between pairs of model outputs.

4.4.1 Pairwise Preference Data¶

Annotators are shown a prompt and two responses — one chosen (\(y_w\), the winner) and one rejected (\(y_l\), the loser) — and mark which they prefer. Pairwise comparison is easier and more consistent than assigning absolute quality scores.

4.4.2 Bradley-Terry Model¶

The reward model is trained with the Bradley-Terry pairwise ranking model. The probability that response \(y_w\) is preferred over \(y_l\) given prompt \(x\) is:

where \(r(x, y)\) is the scalar reward and \(\sigma\) is the sigmoid function. The training objective minimises the negative log-likelihood of observed preferences:

4.5 PPO with KL Penalty¶

Once the reward model is trained, the SFT model is optimised using Proximal Policy Optimisation (PPO).

4.5.1 Setup¶

• Policy \(\pi_\theta\): the language model being optimised, initialised from the SFT model.
• Reference policy \(\pi_{ref}\): a frozen copy of the SFT model.
• Reward signal: \(r(x, y)\) from the reward model.
• Advantage \(\hat{A}_t\): estimated advantage of generating token \(a_t\) at step \(t\).
• Probability ratio: \(r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{ref}(a_t \mid s_t)\).

4.5.2 Objective¶

The KL penalty is essential: without it, the policy quickly learns to exploit the reward model in ways that score highly but produce nonsensical or harmful text — reward over-optimisation.

4.6 Direct Preference Optimisation (DPO)¶

DPO (Rafailov et al., 2023) eliminates the separate reward model and RL loop. It shows that the optimal policy under the RLHF objective has a closed-form relationship to the reference policy, allowing direct supervised optimisation on preference data.

4.6.1 Why DPO Is Simpler¶

4.6.2 DPO Objective¶

Intuitively, the model is rewarded for increasing the log-probability of the preferred response \(y_w\) relative to the reference policy, while decreasing the log-probability of the rejected response \(y_l\). The \(\beta\) hyperparameter controls the trade-off between following preferences and staying close to the reference.

4.7 Constitutional AI¶

Constitutional AI (CAI, Bai et al., 2022), developed by Anthropic, reduces dependence on human preference labellers by using the model itself to generate preference labels — a technique called RLAIF (Reinforcement Learning from AI Feedback).

4.7.1 The Constitution¶

A constitution is a set of principles in natural language that define good behaviour:

1. Prefer responses that do not assist in creating weapons.
2. Prefer responses that do not deceive or manipulate the user.
3. Prefer responses that are honest about being an AI.
4. Prefer responses that support human oversight of AI.

4.7.2 Stage 1 — Self-Critique and Revision (SL-CAI)¶

The model first generates an initial response to a red-team prompt, then critiques it against a randomly sampled constitutional principle, and finally produces a revised response. The revised responses are used for SFT.

flowchart TD
    A["Red-team prompt"] --> B["Model generates\ninitial response"]
    B --> C["Self-critique:\nwhich principle does this violate?"]
    C --> D["Principle-based revision:\nrewrite to comply"]
    D --> E["SFT on revised responses"]

    style A fill:#dc3545,color:#fff
    style E fill:#198754,color:#fff

4.7.3 Stage 2 — RLAIF¶

The model generates pairs of responses, and a separate "feedback model" (often the same base model) selects which response better satisfies the constitution. These AI-generated preference labels train a reward model that is then used with PPO or DPO.

Scalability advantage: RLAIF can generate millions of preference pairs at the cost of inference compute alone, without human annotators. Humans are only needed to write the constitution and seed the red-team prompts.

4.8 Safety Techniques¶

Alignment encompasses a suite of techniques beyond preference optimisation.

4.8.1 Harmlessness Training¶

Annotators specifically flag harmful outputs — violence, self-harm content, illegal instructions. Preference data includes high-weight comparisons where the chosen response is a refusal or safe redirection, and the rejected response is the harmful content.

4.8.2 Refusal Training¶

Models are trained to refuse requests that cross defined policy lines. Good refusal training produces:

• Appropriate refusals: declining genuinely harmful requests.
• Calibrated acceptance: not refusing benign requests due to superficial keyword matches.
• Helpful redirection: offering safe alternatives when refusing.

Overly aggressive refusal training produces over-refusal — declining to discuss medication dosages, historical violence, or security concepts. The over-refusal rate on benign prompts is a key calibration metric.

4.8.3 Honesty Objectives¶

Models are explicitly trained to:

• Express calibrated uncertainty ("I am not certain, but...").
• Acknowledge the limits of their knowledge rather than hallucinating.
• Not make technically-true-but-misleading statements.
• Acknowledge being an AI when sincerely asked.

4.9 Jailbreaks and Defences¶

A jailbreak is a technique that bypasses a model's safety training to elicit policy-violating outputs.

4.9.1 Attack Taxonomy¶

4.9.2 Many-Shot Jailbreaking in Detail¶

Many-shot jailbreaking exploits the in-context learning capability of long-context models. With 100K+ token context windows, an adversary can provide enough in-context demonstrations of policy violations to overwhelm the RLHF signal. The model pattern-matches to the in-context distribution rather than its trained values.

4.9.3 Current Defences¶

No single defence is sufficient. Production systems layer multiple defences — a defence-in-depth architecture.

4.10 Red-Teaming¶

Red-teaming is the systematic adversarial testing of a model to find safety failures before deployment.

4.10.1 Manual Red-Teaming¶

Human red-teamers — typically domain experts in security, chemistry, psychology, or the application domain — attempt to elicit policy-violating outputs through creative adversarial prompts.

Structured process:

1. Scoping: define harm categories in scope (e.g., bioweapons uplift, CSAM, financial fraud).
2. Attack planning: brainstorm attack vectors for each harm category.
3. Execution: attempt attacks; document successful jailbreaks with exact prompts and responses.
4. Severity triage: rate severity and base rate for each confirmed failure.
5. Mitigation loop: share findings with the safety team; repeat after mitigations are deployed.

4.10.2 Automated Red-Teaming with an Adversarial LLM¶

Automated red-teaming uses an adversarial LLM to generate attack prompts at scale, covering far more of the attack surface than manual red-teaming alone.

flowchart LR
    A["Attacker LLM\n(generates adversarial prompts)"] -->|adversarial prompt| B["Target LLM"]
    B -->|response| C{"Safety Classifier"}
    C -->|unsafe — jailbreak succeeded| D["Log failure\nStore prompt + response"]
    C -->|safe| E["Continue"]
    D -->|reward signal| A
    E -->|next iteration| A

    style A fill:#dc3545,color:#fff
    style D fill:#ff9800,color:#fff

The attacker LLM is trained or prompted to maximise the probability of eliciting a policy-violating response. The classifier labels outputs as safe or unsafe. This loop can execute millions of iterations per day, enabling coverage that manual red-teaming cannot match.

4.11 Evaluation¶

4.11.1 Safety Benchmarks¶

4.11.2 Human Evaluation Protocols¶

Automated benchmarks are necessary but insufficient. Human evaluation involves:

1. Side-by-side comparison: annotators compare two model outputs for a given prompt; the aligned model should be preferred.
2. Absolute rating: annotators rate outputs on helpfulness, harmlessness, and honesty (H³) Likert scales.
3. Adversarial probing: annotators specifically attempt to elicit policy violations and report their success rate.
4. Demographic coverage: evaluation panels must be demographically diverse to catch bias failures invisible to homogeneous evaluator pools.

Exercises¶

1. Specification gaming diagnosis: A reward model scores responses on a 1–5 scale and is known to assign higher scores to responses that are longer and more confidently worded. Describe two specific behaviours a PPO-optimised policy is likely to learn. For each, propose a metric that would detect it in production.

2. DPO training loop: Given a preference dataset of 1,000 (prompt, chosen, rejected) triples stored as JSON Lines, write a complete PyTorch training loop implementing the DPO objective. Include gradient accumulation for effective batch size 64 with micro-batches of 4.

3. Constitutional AI implementation: Write a five-principle Constitution for a legal research assistant. For each principle, write one example of a response that violates it and one that upholds it, formatted as the SL-CAI training format.

4. Red-team planning: You are red-teaming a coding assistant that has access to a code execution sandbox with internet access. List five distinct attack categories, one concrete attack prompt per category, and the potential harm if the attack succeeds.

5. Benchmark selection: A startup is deploying an LLM-powered medical information chatbot. Which three benchmarks from Section 4.11.1 are most relevant, and why? Design one additional domain-specific evaluation that the standard benchmarks do not cover.

Summary¶