This paper proposes a vision-based fabric inspection system for the circular knitting machine. Firstly, a comprehensive fabric database called Fabric Defect Detection Database (FDDD) are constructed. To extract significant features of fabric images, shearlet transform is used. Means and variances are calculated from all subbands and combined into a high-dimensional feature vector. The proposed system is evaluated on a circular knitting machine in a textile factory. The real-time performance analysis is only carried out by inspecting single jersey knitted fabric. Our proposed system achieves the highest accuracy of 94.0% in the detection of single jersey knitting fabric defects.