This research is to explore the relationship between a drivers visual features and driving behaviors of distracted driving, and a random forest (RF) method is developed to classify driving behaviors and improve the accuracy of detecting distracted driving. Drivers were required to complete four distraction tasks while they followed simulated vehicles in the experiment. In data analysis, the features of distracted driving behaviors are first described, and the visual data are classified into three distraction levels based on the AttenD algorithm. Based on the collected data, this paper shows the relationship between visual features and driving behavior. Significant differences are discovered between different distraction tasks and distraction levels. Additionally, driving behavior data is used to build an RF model to classify distracted driving into three levels. Results demonstrate that this model is feasible to capture the classification of distraction and its accuracy for each distraction task is over 90%. Areas under receiver operating characteristic curve calculated through error-correcting output codes are mainly around 0.9, indicating good reliability. With this classification method, distraction levels could be classified with vehicle operation characteristics. The model established by this method could detect distractions in actual driving through the detection of driving behavior without the need of eye tracking systems.