Intelligent vehicles are becoming popular to an ever greater extend. However, there is still a long way to go before fully automated driving can reach a relatively large market penetration. Driving safety in the mid-term future thus increasingly depends on the coordination between the driver and automation. New issues appear. For instance, does the driver have enough time to take over the vehicle control when encountering hazardous scenarios which automation is not able to cope with? Microsoft Kinect is one of the best candidates for monitoring drivers’ position thanks to its innovative feature of real time motion capture without use of markers and its low cost. However, when body parts are partially occluded, the accuracy of human skeleton from Kinect will drop markedly. Inspired by previous researches, the present work focused on testing a data driven approach for improving driver’s upper body movement reconstruction when using a Kinect camera. Firstly, a database of accurately captured driver poses from different motion clips is organized with a structure called Filtered Pose Graph, which indicates the intrinsic correspondence between poses. Secondly, the reliability of individual skeletal joint from Kinect posture is evaluated. Meanwhile, weighted K-Nearest-Neighbors and Principal Component Analysis are applied to extract similar postures from the prior database and to predict driver’s current possible motion clip. Thirdly, Simulated Annealing Algorithm is utilized to synthesize natural and kinematically valid posture to help reconstruct unreliable or missing parts of the driver. Finally, Kalman Filter is used to ensure temporal continuity across frames. The preliminary results show that the implemented method can remarkably improve the quality of reconstructed postures and our framework outperforms previous methods in terms of average error and motion continuity.