Economic and environmental considerations have prompted the use of reclaimed asphalt pavement (RAP) and recycled asphalt shingles (RAS) in asphalt mixtures. However, given the concerns about long-term pavement performance, state departments of transportation (DOTs) tend to limit the quantities of these recycled materials unless certain mixture modifications are made [e.g., use of a softer virgin binder performance grade (PG) and warm-mix asphalt technology, the addition of a recycling agent (RA), or any combination of these modifications]. This study focused on the stiffness characterization of recycled asphalt mixtures with combinations of virgin binder PG, RAP, RAS, and RA. Materials were collected from two field projects in Texas and Indiana, and laboratory specimens were prepared and tested for resilient modulus, and dynamic modulus (| E*|) after short-term oven aging (STOA) and long-term oven aging (LTOA). An RA effectiveness parameter was proposed to quantify the rejuvenating effect of RA, which was defined as the percentage reduction in mixture stiffness for the recycled mixture with RA versus the corresponding control mixture without RA. Furthermore, the | E*| test results were analyzed with a Black Space diagram to discriminate asphalt mixtures with different stiffness and relaxation characteristics. The test results indicated that the incorporation of RA was effective in reducing the stiffness of asphalt mixtures with high recycled material content, but the effectiveness diminished with aging. Moreover, recycled mixtures with a softer and less brittle virgin binder and an RA at a higher dosage showed desirable stiffness and relaxation properties after STOA and LTOA.