While social interaction and play in a VR environment are becoming ever more popular, little is known about how social VR games affect users. The purpose of this study was to elucidate the role of several contingent factors in social VR games by modeling the relationships between involvement, well-being, depression, self-esteem, and social connectedness. A conditional process-moderated mediation model of the measured variables was analyzed with 220 pieces of collected data. The result showed that: (1) the direct effect of involvement on well-being was significant, and (2) the index of moderated mediation involving depression, self-esteem, and social connectedness was significant. We conclude that high levels of involvement in social VR games by socially isolated users with low self-esteem can negatively affect their well-being. The findings of this study contribute in several ways to our understanding of the effect of social VR games upon users and provide important practical implications.