This study evaluated the effect of different aggregate sources along with their Corresponding change in mixture characteristics to determine the influence of both on binder oxidation rates and changes in mixture stiffness when compacted mixtures were exposed to laboratory aging conditions. The two aggregate sources, Colorado and Nevada, had different gradations and different water absorption rates, which led to differences in the calculated asphalt binder apparent film thicknesses (AFT) for each mixture. Two asphalt binders, an unmodified PG 64-22 and a styrenebutadienestyrenemodified PG 64-28, were used. The overall findings of the study indicated that both the aggregate and mixture characteristics influenced the oxidation rates of the binder, with the two binders oxidizing by similar amounts when aged in mixtures with the same characteristics (AFT and mixture air voids). The oxidation changes in the binder had differing effects on the stiffness of the mixture as a function of age. Not only were the aggregate and mixture characteristics important to the mixture stiffness and aging relationship, but the binder characteristics themselves, in particular polymer modification, influenced the aging and stiffness relationships of mixtures with age.