We use both analytical and numerical approaches based on the Nambu-JonaLasino (NJL) or the Polyakov-Nambu-Jona-Lasino (PNJL) model to investigate the spinodal instability of a baryon-rich quark matter. In the analytical approach, we obtain via the linear response theory the boundaries of the spinodal region and calculate the growth rate of unstable modes during the early stage. We find that at the mean-field level, the boundaries of spinodal instabilities shrink with the wave number of unstable modes. Both the vector interaction and quantum effect suppress the spinodal instability. And the critical temperature Tc in the PNJL model is almost twice as large as in the NJL model. The collisional effects are then studied by solving the linearized Boltzmann equation, and are found to reduce the growth rate of unstable modes by an amount given by the inverse of the relaxation time. Numerically, we solve the transport equations by the test-particle method and study the phase separation for a quark matter in both a static box and an expanding fireball. We have calculated the higher-order density moments of the quark matter in a box and found them to increase and saturate at large values after phase separation, making them possible signals for a first-order phase transition in baryon-rich quark matter. The skewness of the quark number event-by-event distribution in a small sub-volume of the system is also found to increase, but this feature disappears if the subsystem is large. In the expanding quark matter, we find the expansions to be slowed by the presence of a first-order phase transition. Also, density clumps are found to appear and lead to an anisotropy in the momentum space, which can be characterized by the scaled density moments and the anisotropic elliptic (v2) and quadrupolar (v4) flows. An enhancement in the dilepton yield is also observed.
