Due to growing interest in automated driving, the need for better understanding of human driving behavior with large uncertainties have further increased, such as driving behavior at crossroad and roundabout. Driving behavior at roundabout is greatly influenced by different dynamic factors such as speed, distance and the heaviness of the potentially conflicting vehicles, and drivers have to choose whether or not to leave at the upcoming exit or stay according to these dynamic factors. In this paper, the influential dynamic factors and driving behavior characteristics at the roundabout is analyzed in detail, a novel random forest based method is then proposed to predict driver behavior. For training the prediction model, four typical roundabout layouts were created under a real-time driving simulator with PanoSim-RT and dSPACE. Traffic participants with different motion style were also set in the simulation platform to mimic real driving conditions. Twenty drivers under different age groups were chosen for the data acquisition. Samples of these drivers were used in training the random forest classifier. The accuracy of predictions for test dataset indicates that random forest classifier has good performance in predicting the roundabout behaviors of human drivers.