To understand the potential risks of heavy metals, including their bioavailability and toxicity, 15 surface sediment samples were collected from Luhun Reservoir in Luoyang city, China. Total concentrations and chemical fractions of Cd, Cr, Cu, Ni, Pb, and Zn were analyzed. Various rating methods were used to evaluate the degree, risk, and toxicity of the heavy metal pollution. Results showed that Cd and Pb were preferentially associated with exchangeable (55.77–69.76%) and reducible (53.54–69.43%) fractions, respectively, and therefore exhibited high potential availability. Cr (57.14–86.56%) and Ni (32.21–72.77%) occurred primarily in the residual fraction. Metal concentrations in the effective fraction of the sediment decreased in the order: Cd (96.32%) > Pb (91.61%) > Cu (64.54%) > Zn (57.23%) > Ni (41.51%) > Cr (21.68%). Risk assessment indicated that the risk for Cd is extremely high (62.96%); Cu, Pb, and Zn are ranked as medium risk. Based on the potential ecological risk index, these metals (especially Cd) showed higher potential risk near the dam region. Toxic unit values (2.89–6.05) in more than 60% of sediment sites exceeded a value of 4, and Pb had a relatively higher contribution (1.06–2.65). Cd and Pb are the main contaminants in sediments of Luhun Reservoir and should be paid more attention in the future.