Based on the coordination principle of Lewis acids, a 4-mercaptophenylboronic acid (4-MPBA)-modified novel dumbbell-shaped Au–Ag nanorod (4-MPBA@DS Au–AgNR) substrate was developed, which could be combined with the surface-enhanced Raman scattering (SERS) technique to detect SO₄²⁻ with high sensitivity and specificity. DS Au–AgNRs synthesized in this study with a dumbbell-shaped structure were verified by finite-difference time domain (FDTD) simulation to be capable of stimulating strong localized electromagnetic enhancement (EM) at nano-edge and gap, generating a large number of “hot spots” exhibiting excellent SERS performance. The 4-MPBA modified on its surface could specifically recognize SO₄²⁻, producing a change in the spectral peak at 1382 cm⁻¹, thus realizing highly sensitive and specific sensing of SO₄²⁻. Under optimized conditions, this SERS sensor responded rapidly to SO₄²⁻ within 2 minutes and demonstrated outstanding specificity. Calculation of the ratio of the characteristic peaks at 1382 and 1070 cm⁻¹ (I₁₃₈₂/I₁₀₇₀) enabled the quantitative detection of SO₄²⁻ in the range of 1 × 10⁻⁸–1 × 10⁻³ M, and the detection threshold was as low as 1 nM, which was superior to those of similar detection methods. Importantly, the utility and reliability of this SERS substrate for the determination of SO₄²⁻ in actual samples were evaluated using ion chromatography as the gold standard, and there was no significant difference between the two protocols (P > 0.05), and the RSD was less than 6% with a satisfactory recovery rate (97.6–102.3%). Therefore, the present protocol has the advantages of simplicity and rapidity, high sensitivity, specificity, stability, and practicability in the determination of SO₄²⁻ in aqueous solution, providing a reliable solution for tracing SO₄²⁻ in the fields of food safety and environmental testing.