Agricultural nonpoint source pollution (NPS) is the main water-use impairment in the upper watershed of the Miyun Reservoir in Beijing, China. Selection and placement of best management practices (BMPs) in heterogeneous watersheds, requires a multi-objective optimization framework to identify the most cost-effective conservation strategy to achieve desired water quality goals. In this paper, a novel optimization methodology was developed, utilizing a BMP database that includes BMP reduction efficiencies and costs, using a multi-objective sorting genetic algorithm (NSGA-II, nondominated sorting genetic algorithm-II) combined with the Soil Water and Assessment Tool (SWAT) served as the NPS watershed model. Cost-effectiveness curves (optimal fronts) between pollutant reduction and total net cost input were obtained for the upper watershed of Miyun Reservoir. The optimal combination of BMP, which include a combination of conservation tillage, careful timing of 30% less fertilizer application, contour planting, and use of a 10-m edge-of-field buffer strip, indicate that the least costly scenario reduced total nitrogen (TN) and total phosphorus (TP) loads by 33% at a cost of 1.02 × 10⁶ China Yuan. The cost-effective scenario reduced TN and TP loads 44% and 68% at a cost of 2.52 × 10⁷ and 5.64 × 10⁷China Yuan. The greatest reduction scenario reduced TN and TP loads 55% and 76%, respectively, at a cost of 2.01 × 10⁸ and 2.48 × 10⁸ China Yuan. Watershed with poultry operations, required a 30% reduction in number of birds, along with a 30% reduction in the amount of manure applied was needed to achieve water quality goals. Use of the coupled BMP optimization model can assist the policy makers achieve a cost-effective implementation of best management practices to mitigate agricultural nonpoint sources at a watershed scale.