Operational modal analysis based on power spectral density transmissibility functions (PSDT) is a powerful tool to identify the modal parameters with low sensitivity to excitations. For pole extraction from the PSDT function, a proper parametric identification method such as the Poly-reference Least squares Complex Frequency-domain method (PLSCF) or poly-Max method can be used. Then, the system poles can be identified from a Stabilization Diagram (SD) with overestimating the system model order. In this way, the spurious modes can be incorrectly identified which these modes should be distinguished and removed from the system poles. Due to several sources,the rotor systems may be have the asymmetric damping or stiffness matrices that this form of modal matrices can increases the complexities of identification procedure. To overcome this problem, in this paper, a new algorithm is proposed to construct a clear stabilization diagram for modal parameter identification of rotor system based on the PSDT functions. The proposed algorithm is validated by a computer simulation.