Various algorithms such as emergency brake or crash warning using V2X communication have been published recently. For such systems hard real-time constraints have to be satisfied. Therefore latency needs to be minimized to keep the message processing delay below a certain threshold. Existing V2X systems based on the IEEE 1609 and SAE J2735 standards implement most message processing in software. This means the latency of these systems strongly depends on the CPU load as well as the number of incoming messages per time. According to safety constraints all messages of nearby vehicles have to be processed, whereby no prediction of the message importance can be given without analyzing the message content.Regarding the aforementioned requirements we propose a novel architecture that optimizes latency to satisfy the hard real-time constraints for V2X messages. The novel concept defines latencies of the standards that are safety critical for V2X communication and provides mechanisms to reduce the load of the V2X application processing unit (V2X-APU). The concept supports filtering of messages based on position and direction (GeoCast) to reduce the amount of relevant messages. The concept was implemented with dedicated hardware accelerators to support the V2X-APU. This hardware implementation optimizes the data-flow to minimize the latencies in the V2X communication unit.We showed that latencies, processing times and CPU load can be minimized for V2X communication. An evaluation of the results shows that our concept can provide guaranteed small message latencies even in worst case conditions, allowing safety algorithms to work effectively and reliably.