ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

Accepted by ICML2026. Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs) but usually exhibits limited generation diversity due to the over-incentivization of positive rewards. Although methods like Negative Sample Reinforcement (NSR) mitigate this issue by upweighting penalty from negative samples, they may suppress the semantic distributions shared between positive and negative responses. To boost reasoning ability without losing diversity, this paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses. We theoretically link Lazy Likelihood Displacement (LLD) to negative-positive head-gradient interference and derive a single-forward proxy that upper-bounds representation alignment to guide conservative advantage reweighting. ResRL then projects negative-token hidden representations onto an SVD-based low-rank positive subspace and uses projection residuals to modulate negative gradients, improving reasoning while preserving diversity and outperforming strong baselines on average across twelve benchmarks spanning Mathematics, Code, Agent Tasks, and Function Calling. Notably, ResRL surpasses NSR on mathematical reasoning by 9.4% in Avg@16 and 7.0% in Pass@128.

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning (2026)
• Rethinking Exploration in RLVR: From Entropy Regularization to Refinement via Bidirectional Entropy Modulation (2026)
• CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR (2026)
• GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification (2026)
• SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models (2026)
• Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning (2026)
• From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space (2026)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend