Technological Trends

Deliberative alignment

Paper review

The purpose of Deliberative Alignment (DA) is to improve the safety and trustworthiness of large language models (LLMs) in domains where safety is critical, such as healthcare, legal advice, or education. By teaching models to explicitly reason over safety policies before responding to user requests, DA ensures:

1. Adherence to safety standards. Models can reliably follow predefined rules and policies, minimizing harmful or inappropriate responses.
2. Resilience against adversarial attacks. Models become robust to “jailbreak” prompts, where users attempt to trick them into producing disallowed content.
3. Better generalization. They perform well even in unfamiliar or complex situations not covered during training.
4. Improved user experience. By reducing overrefusals (unnecessary denials of legitimate requests), this approach ensures the models are not overly restrictive.

Current LLMs face two major challenges:

1. Instant responses without deliberation. Models typically respond immediately, without adequate reasoning about complex safety scenarios.
2. Implicit learning of safety standards. Models infer safety rules indirectly from large datasets of labeled examples, which limits their ability to generalize.

These limitations can lead to unsafe outputs, unnecessary refusals, or vulnerability to adversarial prompts.

DA introduces a paradigm shift by embedding knowledge of safety policies into the model and training it to:

• Recall relevant policies during interaction.
• Explicitly reason about these policies to produce policy-compliant answers.

This is achieved by incorporating chain-of-thought (CoT) reasoning, where the model explains its thought process as it examines user prompts, determines which policies are applicable, and generates an appropriate response.

The training is carried out in two stages:

1. Supervised fine-tuning (SFT)
   • A dataset is generated where each example includes:
     • A user prompt.
     • Relevant safety policies.
     • A reasoning process (CoT) that explains why a particular policy applies.
     • The final, policy-compliant response.
   • This teaches the model how to reference and reason about policies in its responses.
   • The training examples are created using automated techniques such as context distillation, where the policies are embedded into the training data (e.g., safety guidelines for topics like self-harm, illicit behavior, etc.).
2. Reinforcement learning (RL)
   • A reward model evaluates how well the model adheres to policies.
   • During this stage, the model refines its reasoning ability and becomes better at identifying complex or borderline cases.

A notable example of DA in action is illustrated on page 2 of the document:

• A user attempts to bypass restrictions by encoding a harmful request.
• The model decodes the request, identifies it as a violation of OpenAI’s safety policies, and refuses to comply.
• The reasoning process is visible in the model’s chain-of-thought, showing how it correctly applied the policies.

DA eliminates the need to provide detailed safety policies at runtime (which would increase latency and complexity). Instead, the model learns to recognize when policies are relevant and how to apply them, even when they are not explicitly included in the conversation context.

The key advantages are:

1. Precision in handling disallowed content
   • The model can identify and refuse requests for harmful content more effectively than traditional training methods.
   • For example, the model outperforms GPT-4o in handling requests for disallowed content (e.g., hate speech or illicit advice).
2. Style guidelines compliance
   • The model follows predefined guidelines for refusals, safe completions (e.g., self-harm responses), and compliance.
   • For instance, it provides empathetic and resource-oriented responses to self-harm prompts while refusing to share harmful methods.
3. Robustness to jailbreaks
   • Deliberative Alignment significantly improves the model’s ability to resist adversarial prompts.
   • On the StrongREJECT benchmark, the model achieves high performance even with encoded or multilingual jailbreak attempts.
4. Generalization to out-of-distribution (OOD) scenarios
   • The model performs well on prompts in unfamiliar languages or encoded formats, demonstrating strong generalization capabilities.

The key findings are:

• DA models outperform other leading LLMs (e.g., GPT-4o, Claude 3.5) across multiple benchmarks:
  • Safety. Reducing the rate of unsafe outputs.
  • Overrefusals. Minimizing unnecessary denials of legitimate requests.
  • Jailbreaks. Improving resistance to adversarial attacks.
• For instance, Table 1 on page 8 shows that the o1 model has higher adherence to safety policies and better performance on challenging tasks compared to GPT-4o.

This technology has critical implications for domains where the consequences of unsafe or incorrect outputs can be severe, such as:

• Healthcare. Providing safe and empathetic responses to mental health queries.
• Legal advice. Avoiding the generation of illicit advice while offering general guidance.
• Education. Ensuring compliance with ethical guidelines in sensitive topics.

The scalability of DA, combined with its robustness and nuanced control, makes it a promising approach for the future of safe AI. By directly embedding safety policies into the reasoning process, it allows for better alignment with human values and institutional guidelines.

In conclusion, DA enhances the safety, reliability, and usability of language models by teaching them to reason over safety policies during interactions, ensuring both compliance and adaptability in diverse scenarios. This represents a significant step forward in aligning AI capabilities with human expectations and ethical considerations.

What’s RAG

The coming wave of AI

Google Keynote 2024 and Gemini

The crosstalk between technology and ethics will lead us into a different future