Learning only from a small set of examples remains a huge challenge in machine learning. Despite recent breakthroughs in the applications of neural networks, the applicability of these techniques has been limited by the requirement for large amounts of training data. What’s more, the standard supervised machine learning method does not provide a satisfactory solution for learning new concepts from little data. However, the ability to learn enough information from few samples has been demonstrated in humans. This suggesting that humans may make use of prior knowledge of previously learned model when learning new ones on minimal training examples. In the autonomous driving area, the model learns to drive the vehicle with training data from humans, and most machine learning based control algorithms require training on very large datasets. Collecting and constructing training data set takes huge time and needs specific knowledge to gather relevant information. This paper aims to learn control parameters from only a few training images. We build a simple control system which can use prior knowledge to correct parameters when the vehicle deviates from the training route, and allows for learning on minimal training examples. The system introduces a new architecture, that, when combined with neural networks, significantly lower the amounts of data required to make meaningful predictions and improves the ability to learn meaningful surrounding information over never seen scenario. We test a simple implementation of our algorithm on a 1/10-scale autonomous driving vehicle. The proposed models produce informative control command when the number of training examples is too small for other methods to operate, and recover the lane deviation successfully.