The use of portable devices for in-vehicle environments has become a major cause for driver distraction which can be a contributing factor in crashes of varying intensity. Despite this fact, the number of drivers choosing to use using these devices while driving is increasing rapidly. On the positive side, smart portable devices are equipped with a variety of useful sensors such as cameras, microphones, accelerometer, gyroscope, etc. which could be leveraged to help reduce driver distraction. Careful utilization and delivery of information extracted from these sensors could potentially prove more useful to drivers rather than distracting them. As a proof of concept, using the sensor information available from an off-the-shelf smart portable device, an automatic system is proposed here for driving maneuver recognition and analysis. Driving maneuvers form the basic building blocks of the driver's intent in completing a route. Being able to automatically identify these and understand how they are performed can help assess the current situation of the driver and evaluate variations in driving patterns. An accuracy of over 90% is achieved in identifying driving maneuvers solely based on portable device sensor information, which is an absolute increase of 15% compared to using CAN-Bus signals. After identifying maneuvers, further analysis is performed to assess deviations from normal driving patterns. In addition to analyzing the entire maneuver, we also examine maneuvers in shorter segments (1 second) to obtain a finer insight into where and how often the deviations occur. Such an analysis provides valuable information on changes in driving habits, and hence can help build and adapt a more comprehensive driving history record.