Moisture damage in asphalt mixtures is a complex phenomenon involving thermodynamic, chemical, physical, and mechanical processes that contribute to pavement deterioration. The empirical nature of the majority of test methods and the inherent variability of the results are the two primary challenges impeding the reliable characterization and assessment of moisture damage. This paper addresses these two challenges by offering a framework that (a) is based on probabilistic analysis that accounts for the uncertainty of the deterioration process and the inherent variability associated with the experimental data, (b) is based on a fracture mechanics damage model for viscoelastic materials that has successfully been applied to the characterization of moisture damage in asphalt mixtures, and (c) combines the fundamental material properties of the asphalt mixture that can be obtained from reliable tests (e.g., surface energy measurements of the material components and dynamic shear test and relaxation modulus data from the dynamic mechanical analyzer). This framework was applied to analysis of the fine matrices of four asphalt mixtures containing different aggregate materials and fillers. The results are compared with the results obtained by assessment of moisture susceptibility for the full asphalt mixtures by the saturation aging tensile stiffness test.