Highly automated driving systems have a responsibility to keep a vehicle safe even in abnormal conditions such as random or systematic failures. However, creating redundancy in a system to respond to failures increases the cost of the system, and simple redundancy cannot detect systematic failures because some systematic failures occur in each system at the same time. Systematic failures in automated driving systems cannot be verified sufficiently during the development phase due to numerous patterns of parameters input from outside the system.A safety concept based on a “safety sustainer” for highly automated driving systems is proposed. The safety sustainer is designed for keeping a vehicle in a safe state for several seconds if a failure occurs in the system and notifying the driver that the system is in failure mode and requesting the driver to take over control of the vehicle. The safety sustainer is designed to support complex and large-scaled automated driving system by following a simple rule such as “The vehicle will not collide with other objects.” Following the simple rule enables detection of a failure in the system easily. The safety sustainer with the simple rule for automated driving system is designed in detail in this research.The improvement effects of the designed safety sustainer under assumed systematic failure rates were evaluated, and efficient points to improve the failure rates were clarified. Furthermore, a “sustainable time” by the safety sustainer was formulated in terms of sensing range and velocity of the vehicle. The formula of calculating failure rates and sustainable time should be considered during the design phase.