Silicone/phosphor composite, which serve as mechanical protection and light conversion material, is an integral part of white light-emitting diode (LED) package. In this paper, a comprehensive study is conducted to investigate the effect of humidity and phosphor on moisture absorption, hygroscopic swelling, mechanical behavior, as well as thermal properties of silicone/phosphor composite in comparison with the pure silicone. SEM/EDAX and FTIR were performed to identify the phosphor and silicone compositions. Through moisture sorption test, it has been observed that the addition of phosphor significantly lowers the capacity of moisture absorption, but accelerates the diffusivity of moisture absorption. The hygroscopic swelling test showed that the phosphor has little effect on the swelling compared to the pure silicone sample. Both moisture absorption/desorption and hygroscopic swelling/de-swelling are reversible. Strain ramp test revealed that the phosphor enhances the stiffness of the composite. The moisture absorption, however, has negligible impact on mechanical stiffness for both pure and composite samples. Finally, thermal expansion test showed that the coefficient of thermal expansion does not change with the addition of phosphor into pure silicone.