Loss of crop diversity has been recognized as a serious threat to agricultural sustainability. This paper considers crop diversity to be a special input in the production process of agricultural goods. Quantile regression was applied to analyze the effects of crop diversity on agricultural output, non-point source pollution and Agricultural Eco-Efficiency (AEE) in China. We found that crop diversity had an inverted U-shaped impact on agricultural output and non-point source pollution, and an overall positive effect on AEE nationally. Crop diversity had positive effects on AEE in the Eastern and Central regions but some inhibitory effects in the Northeast (where cropping is less diverse) and West of China. Under rainfall stress, AEE increased with more crop diversity in Northeast China, but the effects were reversed elsewhere. To promote AEE and sustainable agricultural development, crop diversification should be reduced in Northeast and Western China, but more so in Eastern and Central China.