This paper presents a model for predicting the damage-induced mechanical response of particle-reinforced composites. The modeling includes the effects of matrix viscoelasticity and fracture, both within the matrix and along the boundaries between matrix and rigid particles. Because of these inhomogeneities, the analysis is performed using the finite element method. Interface fracture is predicted by using a nonlinear viscoelastic cohesive zone model. Rate-dependent viscoelastic behavior of the matrix material and cohesive zone is incorporated by utilizing a numerical time-incrementalized algorithm. The proposed modeling approach can be successfully employed for numerous types of solid media that exhibit matrix viscoelasticity and complex damage evolution characteristics within the matrix as well as along the matrix-particle boundaries. Computational results are given for various asphalt concrete mixtures. Simulation results demonstrate that each model parameter and design variable significantly influences the mechanical behavior of the mixture.