Rutting and cracking occur simultaneously in asphalt mixtures as observed in the field and in the laboratory. Existing mechanical models have not properly addressed viscoelastic and viscoplastic deformation together with cracking attributable to model deficiencies, parameter calibration, and numerical inefficiency. This study developed viscoelasticplastic–fracture (VEPF) models for the characterization of viscoelasticity by Prony model and viscoplasticity by Perzyna’s flow rule with a generalized Drucker–Prager yield surface and a nonassociated plastic potential. Viscofracture damage was modeled by a viscoelastic Griffith criterion and a pseudo J-integral Paris’s law for crack initiation and propagation, respectively. The VEPF models were implemented in a finite element program by using a weak form partial differential equation modeling technique without the need for programming user-defined material subroutines. Model parameters were derived from fundamental material properties by using dynamic modulus, strength, and repeated load tests. Simulations indicated that the viscoelastic–viscoplastic–viscofracture characteristics were effectively modeled by the VEPF models for asphalt mixtures at different confinements and temperatures. An asphalt mixture under monotonic compressive loads exhibited a sequenced process including a pure viscoelastic deformation stage, a coupled viscoelastic–viscoplastic deformation stage, a viscoelastic–viscoplastic deformation coupled with a viscofracture initiation and a propagation stage, and then a viscoelastic–viscofracture rupture stage with saturated viscoplastic deformation. The asphalt mixture under repeated loads yielded an increasing viscoplastic strain at an increasing rate during the first half of the haversine load, while the increment of the viscoplastic strain (per load cycle) decreased with load cycles. The finite element program, which is based on a partial differential equation, effectively modeled the coupled viscoelastic–viscoplastic–viscofracture behaviors of the asphalt mixtures.