Impact toughness (or resistance to fracture) is a key material property for press hardened steel used in construction of the safety-critical elements of automotive body structures. Prior austenite grain size, as primarily controlled by the incoming microstructure and austenitization process, is a key microstructural feature that influences the impact toughness of press hardened steel. In this paper, a special Charpy V-notch impact test is developed to quantify the impact toughness of press hardened steel sheets with various prior austenite grain sizes, by stacking a number of thin sheets via mechanical riveting. Both the ductile-to-brittle transition temperature and upper shelf energy are analyzed in an effort to establish a correlation between impact toughness and prior austenite grain size. Within tested conditions, impact performance shows only a slight decrease as the prior austenitic grain size increases from 18 to 38 microns. On the other hand, grain refinement via micro-alloying in the new 1.8 GPa press hardening steel can significantly increase the impact toughness.