Document-Grounded Coaching Agent for Video Skill Assessment

Abstract

This paper presents a document-grounded coaching framework that evaluates learner behavior in videos using instructional materials (e.g., books and manuals) as normative references, without relying on expert reference videos or additional supervised training. The framework first structures document content into an ordered rubric of procedural steps and step-specific evaluation norms. It then estimates step intervals in learner videos under ordering constraints and computes time-wise deviation scores from the corresponding norms. Experiments on 280 basketball practice videos from Ego-Exo4D, paired with a basketball instructional manual, show that the proposed framework provides more accurate step-interval estimation than representative baseline methods. The estimated deviation scores also provide clearer separation between beginners and experts, and qualitative analysis indicates consistency with human judgments of norm violations. These results suggest that document-grounded coaching is a practical and interpretable framework for capturing both temporal structure and execution quality of skill performance.