This paper proposes an image-based vehicle safety system, which has the ability to monitor the driver's status during their driving in all environment conditions. The image process technique, used to detect the driver's face by AdaBoost-based algorithm, is suitable to be applied under the internal environments of cabin. By analyzing the position of driver's face, once the driver's face disappeared in a reasonable area, the system will regard it as inattentive driving. When the acts of inattentive driving occurred, the system will generate warning signals through a buzzer or other devices to draw the driver's attention to notice on his driving. This system has been implemented on low-cost DSP platform and it can be installed efficiently on varied kind of vehicles. Under day and night driving conditions with series of specific viewing angles (from ahead, left/right-turn and turn-back views, which are corresponded to the angles in 0°~±90° and 180°), the averaged correct warning rate is 92.1%. In addition, under the circumstance of attentive driving, the correct warning rate is 96.08%.