The monitoring and diagnosis of structures using output-only ambient vibration tests are becoming increasingly popular nowadays. In this Article a new statistics-based approach is proposed for damage detection and diagnosis (localization and quantification) in structures under their normal working conditions. The proposed approach requires vibration data relative to the current and reference states of the structure as well as a finite element model parameterized using specified damage parameters. Such parameters may include physical and/or geometrical properties of the structure. Modal filtering, combined with an asymptotic local approach, is used to compute an improved residual which turns out to be normally distributed. The mean, covariance and sensitivity of this residual allow the detection of damage based on generalized log-likelihood ratio (GLR) tests. The diagnosis of damage is made possible by applying sensitivity and rejection tests to the residual vector computed directly from the finite element model. Constraints on the damage parameters are also incorporated into these test statistics to ensure that only physically feasible damage is considered. Moreover, in some cases of multiple damage, a branch-and-bound scheme may be necessary to expedite the search for damage. The proposed damage detection and diagnosis algorithm is tested on simulation examples of simple structures.