Roughness is an attribute often used by customers to describe one of the annoying properties of automotive powertrain sounds. However, subjective ratings of roughness do not always correlate well with the psychoacoustic unit of roughness, aspers. The discrepancy between many existing objective measures of roughness and experimentally measured perceived roughness of powertrain noise is what motivates the work described in this paper. Several subjective experiments are conducted with real powertrain sounds as stimuli to study this discrepancy. Objective measures of roughness based on an AM model as well as an auditory model of roughness are applied. The results indicate that loudness can be a confounding factor. Also, contrary to what the objective measures predicted, subjects showed poor performance in these roughness discrimination tasks. The effect of binaural cues is also investigated and test results indicate that it is not a major factor in the scaling of roughness. We conclude with a discussion on the weaknesses of the two types of roughness measures used in the experiments.