In this work, similar (2A12) and dissimilar (6061) aluminum alloy sheets are validly joined using self-piercing rivet process. A quasi-static experiment is proposed to investigate the mechanical behaviors, failures mode, and mechanism of the different joints. Moreover, a method based on deep learning algorithm is anticipated to detect the appearance defects of the SPR welded joints. The results indicated that 2A12 joints of similar sheets contained the advantageous static strength and 6061 similar sheet joints had superior anti-vibration performance conducts. The joints with 6061-2A12 sheets introduced the most decent and comprehensive mechanical properties. The main failure mode of 2A12 similar sheet joints was substrate fracture. The performance of the substrate affects the failure mode of the joint and the plasticity of the substrate is better. When the time comes, the failure mode is mostly pull-off failure. Poor plasticity of the substrate can easily lead to substrate breakage. The reason for joint pull-off and button fall-off failure is that there is large plastic deformation in the lower plate of the joint and the mechanical internal locking structure is damaged. 2A12 substrate breakage belongs to a composite fracture that combines intergranular fracture and microvoid aggregation type fracture. The area of the 6061 substrate near the edge of the sample is shear fracture and the area near the center of the sample thickness is dominated by microvoid aggregation type normal fracture. The effectiveness of the method was verified by conducting a series of experiments and the detection accuracy of the method can reach about 90%. The detection speed was as high as 50 frames per second (FPS), which can effectively solve the problem that the rivet quality was difficult to monitor.