In this paper, a sensor fusion approach is introduced to estimate lane departure. The proposed algorithm combines the camera and inertial navigation sensor data with the vehicle dynamics to estimate the vehicle path and the lane departure time. The lane path and vehicle path are estimated by using extended Kalman filters. This algorithm can be used to provide early warning for lane departure in order to increase driving safety. Additionally, the algorithm can be used to reduce the latency of information embedded in the controls, so that the vehicle lateral control performance can be significantly improved during lane keeping in Advanced Driver Assistance Systems (ADAS) or autonomous vehicles. Furthermore, it improves lane detection reliability in situations when camera fails to detect lanes. Several scenarios are simulated in order to show the effectiveness of the proposed algorithm.