Real-time detection of fasteners is of great significance for improving the inspection efficiency of railway tracks. However, the task is also challenging due to the limited memory and processing capacity of the embedded maintenance systems. To address these challenges, in this paper, we propose a new detection network architecture called MYOLOv3-Tiny, in which the depthwise and pointwise convolution are used to reduce the computational complexity of the network, and the linear bottleneck structure with inverted residual is employed to design the backbone network for efficiently extracting the fasteners’ features. The MYOLOv3-Tiny is trained and tested on the high-resolution track fastener data collected from the Ring Test Field of National Railway Test Center of China. The extensive experiments show the proposed MYOLOv3-Tiny achieves higher detection precision, lower computational complexity and memory consumption, and faster detection speed compared to state-of-the-art methods. Specifically, its detection precision reaches 99.32% after introducing visually coherent image mixup enhancement technology, and the memory consumption is reduced by 43% compared to the YOLOv3-Tiny network with a high detection speed guarantee. Hence, the proposed MYOLOv3-Tiny has the potential to lay a foundation for the real-time detection of railway track fasteners and to be applied to large-scale fastener detection devices.