The objectives of this study were a) to determine how expert judges categorized valid Integrated Vehicle-Based Safety Systems (IVBSS) Forward Collision Warning (FCW) events from review of naturalistic driving data; and b) to determine how consistent these categorizations were across the judges working in pairs. FCW event data were gathered from 108 drivers who drove instrumented vehicles for 6 weeks each. The data included video of the driver and road scene ahead, beside, and behind the vehicle; audio of the FCW alert onset; and engineering data such as speed and braking applications. Six automotive safety experts examined 197 ‘valid’ (i.e., conditions met design intent) FCW events and categorized each according to a taxonomy of primary contributing factors. Results indicated that of these valid FCW events, between 55% and 73% could be considered ‘nuisance alerts’ by the driver. These were the FCW alerts presented in benign conditions (e.g., lead-vehicle turning) or as a result of deliberate driver action (aggressive driving). Only 16% of the FCW alerts were attributed to driver distraction and all of these cases involved a driver looking away from the road scene at an inopportune time. The consistency or agreement in categorization performance of the 6 experienced safety professionals was also examined. Working in pairs, there was either majority or unanimous agreement on 88% of the 197 FCW cases reviewed by the three pairs of judges working independently from one another.