Reinforcement learning large language models
The landscape of reinforcement learning (RL) for large language models (LLMs) has been evolving rapidly with the introduction of innovative methodologies. Notable successes, such as OpenAI’s o1 series and DeepSeek-R1, have showcased the power of large-scale RL in enhancing reasoning capabilities of LLMs.
However, the training processes for these models often remain shrouded in technical intricacies, and the challenge of effectively scaling these methods across diverse domains persists. Recent efforts by the Kwaipilot team at Kuaishou have brought forward a new approach: Two-Staged history-Resampling Policy Optimization (SRPO). This method aims to address these challenges, demonstrating potential for substantial improvements in training efficiency and model performance (Wikipedia, Reinforcement Learning, 2023).
Reinforcement Learning Training Efficiency
Conventional Reinforcement Learning from Preference Optimization (GRPO) faces specific hurdles, particularly in cross-domain generalization. The core issues include performance bottlenecks, inefficient sample utilization, and difficulties in nurturing specialized reasoning skills, especially when dealing with mixed-domain datasets.
Such challenges hinder the scalability of RL methods for LLMs, creating a need for more sophisticated and adaptable training techniques, including large language models applications, including training efficiency applications. To tackle these limitations, the SRPO framework introduces a structured approach to training, focusing on coherent integration of reasoning skills across domains. The SRPO method has proven successful in achieving DeepSeek-R1 – Zero-level performance in both mathematical and code domains concurrently, a first in the field, especially regarding large language models.
This achievement is particularly notable because it requires only one-tenth of the training steps compared to the traditional R1-Zero, signaling a significant leap in training efficiency (arXiv, SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM, 2023).
Reinforcement learning optimization
The Kwaipilot team’s exploration of standard GRPO revealed several bottlenecks that needed to be addressed. Among these were cross-domain optimization conflicts, reduced training efficiency due to similar group rewards, and premature performance saturation.
These issues led to suboptimal performance, primarily because of the inherent differences between mathematical and code data. To resolve these conflicts, a two-stage training paradigm was employed, including reinforcement learning applications in the context of large language models in the context of reinforcement learning, especially regarding large language models, particularly in training efficiency. In the initial stage, the model focuses on mathematical data to build foundational reasoning capabilities.
This phase encourages behaviors such as reflective pausing and step-by – step decomposition. The second stage introduces code data, aiming to enhance coding abilities while reinforcing procedural thinking and recursion skills.
This strategic approach allows the model to integrate and refine its reasoning abilities across domains, resulting in superior performance in both mathematical and programming tasks.
Training Strategies and Reasoning Abilities
An analysis of different training strategies revealed distinct impacts on response length and benchmark performance. Mixed training of math and code data resulted in limited growth, while math-only training fostered strong, generalizable reasoning abilities.
Conversely, code-only training improved performance on code benchmarks but lacked the development of explicit reasoning behavior, particularly in reinforcement learning in the context of large language models, especially regarding training efficiency. The two-stage training approach of the Kwaipilot team emerged as the most effective. It consistently produced detailed reasoning patterns for both math and programming tasks.
Notably, the model developed the ability to spontaneously use code to assist in mathematical reasoning, highlighting the efficacy of the staged training approach in cultivating complex reasoning behaviors.
Training efficiency reinforcement learning
To further optimize training efficiency, the Kwaipilot team introduced History Resampling. This technique addresses inefficiencies observed during the mid-to – late stages of training, where a significant portion of sampled groups produced identical rewards.
History Resampling involves recording reward outcomes and reconstructing datasets for subsequent epochs, especially regarding reinforcement learning, particularly in large language models. The approach focuses on filtering out overly simple samples and retaining those with diverse outcomes, ensuring effective gradient signals. This strategy aligns with curriculum learning principles, gradually exposing the model to increasingly challenging samples.
Compared to Dynamic Sampling methods, History Resampling has demonstrated superior computational efficiency and stable response length growth.
Data quality in training LLMs
Data quality plays a crucial role in the success of training LLMs. The Kwaipilot team meticulously cleaned and filtered their datasets, ensuring the removal of irrelevant and noisy data.
This process involved verifying the correctness of math and code problems, categorizing them by difficulty, and discarding problems with ambiguous or incorrect solutions in the context of reinforcement learning in the context of large language models in the context of training efficiency. Experimental results from the SRPO method indicate a clear improvement in both mathematical and coding abilities. The training process, as illustrated by reward and response length curves, shows stable growth in both areas.
Furthermore, the emergence of reflective reasoning behaviors, such as self-verification and procedural thinking, underscores the adaptive capabilities of the model during RL training.
Reinforcement learning training efficiency
The advancements brought about by the SRPO framework represent a significant step forward in the field of reinforcement learning for LLMs. By addressing conventional challenges and introducing innovative training strategies, the Kwaipilot team has paved the way for more efficient and effective model training across domains in the context of large language models, particularly in training efficiency.
As the field continues to evolve, the integration of sophisticated reasoning capabilities into LLMs holds promise for a wide array of applications, from complex problem-solving to advanced code generation.
