The progress of GPU (Graphics Processing Unit) technology opens a new avenue for boosting computing power. This work is an attempt to exploit the GPU for accelerating VLSI circuit optimization. We propose GPU-based parallel computing techniques and apply them on simultaneous gate sizing and threshold voltage assignment, which is a popular method for VLSI performance and power optimization. These techniques include efficient task scheduling and memory organization, all of which are aimed to fully utilize the advantages of GPUs. Compared to conventional sequential computation, our techniques can provide up to 56 (39 on average) speedup without any sacrifice on solution quality.