Accuracy in detecting a moving object is critical to autonomous driving or advanced driver assistance systems. By including the object classification from multiple sensor detections, the model of the object or environment can be identified more accurately. The critical parameters involved in improving the accuracy are the size and speed of the moving object. In a laboratory experiment, we used three different type of sensor, a digital camera with 8 megapixel resolution, a LIDAR with 40m range, and an ultrasonic distance transducer sensor to identify the object in real-time. The moving object that is to be detected was set in motion at different speeds in the transverse direction to the vehicle (sensor). The size of the moving object was also varied. All sensor data were processed on a real-time prototyping microcontroller. All sensor data were used to define a composite object representation so that it could be used for the class information in the core object’s description. Camera image data from subsequent frames along the time axis are considered in conjunction with the speed and size of the object for developing better recognition algorithms. This composite data was then used by a deep learning network for complete perception fusion in order to solve the detection and tracking of moving objects problem. The laboratory experiments show encouraging results where the proposed deep learning based sensor fusion was used for the moving object detection.