OCSD contrasts structurally matched replay views with and without future observations to isolate observation-specific supervision. It uses the resulting residual to modulate token-level GRPO updates at high-uncertainty steps while preserving the trajectory-level update direction.
Publication
Preprint available as arXiv:2608.04788. The first version was submitted on August 5, 2026.
Overview
On-policy self-distillation can mix support from privileged future observations with token-score changes caused by the replay scaffold itself. OCSD compares Full and Observation-Ablated replay views to discount shared scaffold effects. The paper reports experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales, as well as diagnostic analyses of alignment with local environment feedback.
Tasks and Configuration
- Tasks: ALFWorld, WebShop, and Search-QA.
- Model configurations: Qwen3 1.7B, 4B, and 8B.
- Code license: Apache License 2.0.