OCSD contrasts structurally matched replay views with and without future observations to isolate observation-specific supervision. It uses the resulting residual to modulate token-level GRPO updates at high-uncertainty steps while preserving the trajectory-level update direction.

Publication

Preprint available as arXiv:2608.04788. The first version was submitted on August 5, 2026.

Overview

On-policy self-distillation can mix support from privileged future observations with token-score changes caused by the replay scaffold itself. OCSD compares Full and Observation-Ablated replay views to discount shared scaffold effects. The paper reports experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales, as well as diagnostic analyses of alignment with local environment feedback.

Tasks and Configuration

  • Tasks: ALFWorld, WebShop, and Search-QA.
  • Model configurations: Qwen3 1.7B, 4B, and 8B.
  • Code license: Apache License 2.0.

Resources