There is great interest in transition metal-doped InSe because of its high nonlinearity and ultrafast response time at higher light fluence. Herein, Ag-doped InSe nanofilms were precisely manufactured using a direct current-radio frequency sputtering method, and their ultrafast broadband nonlinear optical responses in near-infrared were systematically researched. Ag-doped InSe nanofilm exhibited a broadband nonlinear optical response (800–1100 nm) and ultrafast carrier absorption (<1 ps), and can act as a potential semiconducting material for all-optical devices. Through precise control of the sputtering process parameters, Ag-doped InSe nanofilms were successfully prepared that were smooth, uniform, and exhibited no cracks. Nonlinear optical studies (femtosecond transient absorption spectroscopy and Z-scan measurement) indicated that nonlinear absorption behavior in Ag-doped InSe nanofilm withstands a transformation from saturation absorption to reverse saturation absorption arising from ground state bleaching, free-carrier absorption (FCA), and two-photon absorption (TPA). Additionally, nonlinear refraction behavior in Ag-doped InSe nanofilm was successfully detected near the intrinsic absorption edge, which arose from Kerr refraction and free-carrier refraction. More importantly, the broadband nonlinear response, ultrafast carrier absorption, and carrier recovery time of Ag-doped InSe nanofilm has the ability to controllably tune via Ag doping. Furthermore, Ag-doped InSe nanofilm possesses the nonlinear figure of merit (FOM) of 2.02, which indicates that Ag-doped InSe nanofilm is a promising semiconducting material for all-optical switching devices in near-infrared.