Industrial production often faces a variety of complex working conditions that lead to various defects, including Mura, on the surfaces of various industrial products. We propose a reconstruction network called RecTransformer, which is developed with a transformer for anomaly inpainting. RecTransformer is designed to effectively detect various types of surface defects despite using only a small number of defect samples. RecTransformer simplifies the defect detection problem to a patch-level image completion problem. Without using convolution, the given block image is processed by the transformer model to generate a defect-free reconstructed image. Herein, global semantic information is established, and an attention mechanism is built in the patch sequence, and the spatial information of the patches is determined by position encoding to complete the global image reconstruction process. With a limited number of defect samples as training data, the RecTransformer algorithm accurately reconstructs defects. It achieves an area under the receiver operating characteristic curve score of 97.6% for pixel-level segmentation on the testing dataset. Experiments conducted on a universal surface defect dataset demonstrate the effectiveness of the RecTransformer algorithm. RecTransformer can be adapted to detect various types of surface defects, including Mura in display devices, with only a small number of defect samples.