A systematic analysis method of the mechanism and simulation of self-excited vibration in the transmission is proposed. The fact that there is self-excited vibration with hard excitation characteristics in the vehicle transmission at high slip rate is observed by analyzing the results of the experiments. The cause of the self-excited vibration is found by means of the phase plane analysis and the vibration energy analysis approach. A 21-degree-of-freedom variable parameter model of the vehicle torsional vibration is established to simulate the nonlinear vehicle feedback-control system of the self-excited oscillation. A good agreement is shown by comparing the predicted results with measured ones.