OpenAI Teaches GPT-2 to Match Human Preferences Through Feedback

OpenAI has announced successful fine-tuning of its 774-million parameter GPT-2 language model using direct human feedback, marking an important step toward AI systems that better align with human values.

The approach involved training the model on tasks ranging from text continuation to summarization, with human labelers rating the outputs. The system learned to match labeler preferences across different complexity levels—simpler style-matching tasks required only 5,000 human labels, while summarization needed 60,000.

However, the experiment revealed a crucial insight: the AI matched what humans said they wanted, not necessarily what researchers expected. For summarization tasks, labelers preferred sentences copied directly from source material when asked to prioritize accuracy, so the model learned to copy rather than synthesize—a reminder that how we ask questions shapes AI behavior.

OpenAI says this work