An instantaneous frequency identification method of vibration signal based on linear chirplet transform and Wigner-Ville distribution is presented. This method has an obvious advantage in identifying closely spaced and time-varying frequencies. The matching pursuit algorithm is employed to select optimal chirplets, and a modified version of chirplet transform is presented to estimate nonlinear varying frequencies. Because of the high time resolution, the modified chirplet transform is superior to the original method. The proposed method is applied to time-varying systems with both linear and nonlinear varying stiffness and systems with closely spaced modes. A wavelet-based identification method is simulated to compare with the proposed method, with the comparison results showing that the chirplet-based method is effective and accurate in identifying both time-varying and closely spaced frequencies. A bat echolocation signal is used to verify the effectiveness of the modified chirplet transform. The result shows that it will significantly increase the accuracy of nonlinear frequency trajectory identification.