Natural driving behavior replication is important for both autonomous driving algorithm development and safety evaluation research, but also brings extra difficulties to normal problems of classification and regression. This is due to the large range and high resolution of driving behavior (i.e. 75 mph range with 1 mph resolution for speed selection and at least 720-degree steering wheel angle with 1 degree resolution for typical driving scenarios) and the subtle and complicated feature representations from the perception perspective. To address these challenges the authors propose a tree-based ensemble learning method that features importance-analysis-based selective scanning and a normalized framework for multi-model integration. This approach predicts general driving behavior directly from raw input (i.e. pixels from forward-facing camera, vehicle dynamic states, turn signal, etc.). In comparison with previous machine-learning solutions like convolution neural network, while sharing the similar nature of parallel computation and advantage of dealing with multi-format high-dimensional data, the proposed method has on par statistical accuracy with better bias-variance trade-off and is easier to train and interpret. In contrast to existing analytical driver modeling methods that are limited to decomposed traffic scenarios like highway driving, lane keeping, vehicle following, obstacle avoidance, etc., the authors tackle the general problem of driving behavior prediction without case-dependent constraints or mediated steps of analysis. The driving data for this research was collected in various environments driven by professional test drivers on a 2016 Acura ILX instrumented with LiDAR, cameras, IMU, and GPS. The proposed method of selective scanning forests driving behavior prediction is validated with excellent performance on general speed selection and steering angle prediction against existing metrics such as convolution neural network and random forest.