Complex characteristics of fatigue behavior were evaluated on the basis of test results and their mechanical analyses. The dynamic shear rheometer was used to characterize fundamental linear viscoelastic properties of asphalt binders and mastics. Various dynamic mechanical tests using cylindrical sandasphalt samples mixed with pure binders, mastics, or both were also performed to estimate viscoelastic characteristics and fatigue behavior. To assess the filler effect, two distinctly compositionally different asphalt binders, AAD-1 and AAM-1, and two fillers, limestone and hydrated lime, were selected. Test results were analyzed using viscoelastic theory, a fatigue prediction model based on continuum damage mechanics, and a rheological composite model. The role of fillers in fatigue resistance was quantified, and induced mechanisms due to filler addition were investigated. The effect of hydrated lime, which is highly binder specific, as a filler was further discussed by comparing test results from hydrated lime filler and test results from limestone filler.