This paper presents a pan-tilt sensor fusion platform for activity tracking and fall-detection which can work as a reliable surveillance system with long-term care function. A low cost thermal array sensor and a distance sensor are integrated together as the sensor module. The sensor module is installed on a pan-tilt orienting mechanism with two rotation degrees of freedom to increase the field of view while reducing the number of sensors used on-board. The performance of the sensor test platform is analyzed. The location of the indoor object as well as its size can be estimated based on a novel sensor fusion algorithm. The support vector machine (SVM) based machine learning algorithm is applied for fall detection. The preliminary experiment result shows a 95% accuracy to identify falling action from similar normal indoor activity such as sitting and picking up stuff.