Distinguishing physical modes from mathematical modes in the modal analysis of complex systems, such as full vehicle structures, is a difficult and time-consuming process. The major tools frequently used are stabilization diagrams, mode indicator functions, or modal participation factors. When closely spaced modes are to be identified, the stabilization diagrams and mode indicator functions are no longer effective. Even the reciprocities of mode shapes and modal participation factors cannot be well satisfied to indicate whether a mode is a physical one, when measurement errors are large. To overcome these difficulties, an optimization procedure is developed, whereby physical modes can be sorted out in a given frequency range while the error between measured and synthesized frequency responses is minimized. An optimal subset selection algorithm is used in this procedure. Mode significant values are also introduced to show the relative significance of each mode in contributing to measured frequency responses. This innovative method proves to be effective in aiding the selection of physical modes for full vehicle structures.