Restoring Calibration in RLHF-Aligned Models
Incorporate a calibration regularization term into the RLHF objective function to preserve uncertainty estimates during post-training.
Source paper
Generated by AI. The paper’s authors did not write these.
Incorporate a calibration regularization term into the RLHF objective function to preserve uncertainty estimates during post-training.
Detect semantic contamination in benchmarks by calculating vector embedding similarities between evaluation items and training corpus samples.
Automate adversarial red-teaming by training a separate attacker LLM via reinforcement learning to generate prompts that expose safety policy violations.
Build a high-fidelity sandboxed digital environment and fine-tune large language models specifically on agentic trajectories to determine the theoretical upper bound of their autonomous capabilities and risks.
This report details the development of GPT-4, a large-scale multimodal Transformer model that exhibits human-level performance on various professional benchmarks, such as the Uniform Bar Exam. The authors emphasize the use of predictable scaling laws to estimate model performance from smaller compute budgets and employ Reinforcement Learning from Human Feedback (RLHF) alongside Rule-Based Reward Models (RBRMs) to align the model with safety guidelines.
The model uses a Transformer-based architecture pre-trained on a massive corpus of text and image data to predict the next token. This is followed by a post-training alignment phase using Reinforcement Learning from Human Feedback (RLHF) and auxiliary Rule-Based Reward Models (RBRMs) to steer the model towards helpfulness and safety.