The study reported in this paper presented a multiscale computational model, along with its validation and calibration, to predict the damage-dependent behavior of asphalt mixtures subjected to viscoelastic deformation and cracking. Asphalt mixture is a classic example of a multiphase composite that represents different lengths of scales. The understanding of the mechanical behavior of asphaltic materials has been a challenge to the pavement mechanics community because of the multiple complexities involved: heterogeneity, anisotropy, nonlinear inelasticity, and damage growth in multiple forms. To account for this issue in an accurate and efficient way, the study reported here presented a two-way linked multiscale computational modeling approach. The two-way linked multiscale model had its basis in continuum thermomechanics and was implemented with a finite element formulation. With the unique multiscale linking between scales and the use of the finite element technique, this model could take into account the effects of material heterogeneity, viscoelasticity, and anisotropic damage growth in small-scale mixtures on the overall performance of larger-scale structures. Along with the brief theoretical model formulation, the multiscale model was validated and calibrated through the comparison of the numerical, analytical, and experimental results of three-point bending beam tests of asphalt mixture samples that involved viscoelasticity, mixture heterogeneity, and cohesive zone fracture.