This paper presents a framework that combines experimental techniques and computational methods for modeling the microscopic response of asphalt mixtures subjected to various loading conditions. The basis of this framework is capturing the three-dimensional microstructure of asphalt mixtures with X-ray computed tomography and a sequence of image-processing methods to identify the microstructure components or mixture phases. This microstructure is then converted to a finite element model in which the various phases are represented with constitutive models that describe their mechanical behavior. In this study, the coarse aggregate phase was modeled as a linear elastic material, and the matrix phase (asphalt, fine particles, and air voids) was represented as a thermoviscoelastic, viscoplastic, and damage model. The analysis results showed that the model captured the effects of temperature, rate of loading, repeated loads, and mixture design on the microstructure response. These results demonstrate that the developed framework will help engineers and researchers to understand the effects of mixture design and material properties on performance and to establish the link between microscopic response and macroscopic behavior.