The threshold effect in the relationship between impervious surfaces and water quality has been a focus in past decades but little attention has been paid to how the threshold effect changes during a rapid urbanization period. This study reveals the temporal variation of threshold effect in the relationship between the percentage of total impervious area (PTIA) and water quality indicators in a reticular river network area in Shanghai, China. The PTIA was surveyed and defined using the ISC method (impervious surface coefficients). A segmented regression model was used to disclose the non-linear relationship between PTIA and water quality. It is confirmed that the threshold effect was different in terms of water quality indicators, but the effect size became smaller as the threshold increased with urbanization level during the period of 1989–2010. Meteorological conditions make influence on the threshold effect, it can be found that the threshold effect is more significant under higher temperature conditions, while in the lower temperature range, there is no significant threshold effect.