In this paper, we present a pipeline and framework for the Intelligent Immersive Learning environment for Programming Robotics Operations (IL-PRO), a novel AI-based approach to assess and enhance learner capabilities in an immersive virtual reality (VR) environment. By integrating telemetry data (both continuous and discrete) and speech data, the IL-PRO pipeline evaluates users’ motor skills and cognitive understanding to deliver personalized, real-time feedback that links their conceptual understanding with motor skill performance. Telemetry data captures precise physical human-system interactions which are processed and analyzed using Machine Learning (ML) tools to capture and rate motor skill capabilities, while speech data is analyzed using Natural Language Processing (NLP) techniques in concert with a Large Language Model (LLM) to simultaneously assess comprehension and task-related knowledge. These insights are then integrated and used to provide feedback and adapt the learning environment dynamically, tailoring tasks and modules to the learner’s specific needs and progress. To demonstrate the feasibility of this approach, we apply the pipeline to a VR task focused on robot acceleration, which emphasizes how motor skills and cognitive understanding work together when learning about inertia in industrial robotic arms. This use case illustrates the pipeline’s comprehensive workflow: data collection, multimodal processing of telemetry and speech using machine learning and AI, integration of cognitive and physical insights, and generation of adaptive, real-time feedback. The IL-PRO pipeline framework advances the development of immersive learning systems, enables research on how users combine motor skills with cognition, and enhances skill acquisition in applied training contexts such as robotics.
