Gear fault diagnosis is important in the vibration monitoring of any rotating machine. When a localized fault occurs in gears, the vibration signals always display non-stationary behavior. In early stage of gear failure, the gear mesh frequency (GMF) contains very little energy and is often overwhelmed by noise and higher-level macro-structural vibrations. An effective signal processing method would be necessary to remove such corrupting noise and interference. This paper presents the value of optimal wavelet function for early detection of faulty gear. The Envelope Detection (ED) and the Energy Operator are used for gear fault diagnosis as common techniques with and without the proposed optimal wavelet to verify the effectiveness of the optimal wavelet function. Kurtosis values are determined for the previous techniques as an indicator parameter for the ability of early gear fault detection. The comparative study is applied to real vibration signals. First, to eliminate the frequency associated with interferential vibrations, the vibration signal is filtered with a band-pass filter determined by a Morlet wavelet whose parameters are optimized based on maximum Kurtosis. Then, to further reduce the residual in-band noise and highlight the periodic impulsive feature, an envelope analysis enhancement algorithm is applied to the filtered signal. The test stand is equipped with three dynamometers; the input dynamometer serves as the internal combustion engine, the output dynamometers introduce the load on the output joint shaft flanges. The gearbox used for experimental measurements is the type most commonly used in modern small to mid-sized passenger cars with transversely mounted powertrain and front wheel drive.