The Yumenkou tunnel is located on the left bank of the Yellow River on the border between Shanxi and Shaanxi provinces, China. It is connected to the Longmen Yellow River Bridge of the Menghua railway. Many unstable rock masses have developed on the cliff above the tunnel entrance, threatening the construction and operation of the tunnel and the bridge. Based on a high-precision three-dimensional (3D) digital terrain model obtained by unmanned air vehicle (UAV) photogrammetry, information of the structural surface which controls the stability of the rock mass in the study area was obtained by manual identification and extraction. We combined the UAV imaging with field survey data to obtain the spatial distribution, shape, scale, and potential failure modes of the unstable rock masses. A 3D numerical model of the unstable rock mass and surface morphology was constructed using a 3D discrete element method. The 3D path and impact energy of the moving unstable rock mass were analyzed. Based on the lithology, topography, structural surface, and other factors, three types of potential rock failure modes were identified in the study area: sliding, toppling, and falling. Differential weathering and the structure plane and stress-unloading cracks are two main factors affecting the stability of the unstable rock mass. According to the result of 3D discrete element method, once the unstable rock masses fail, about 50% of which might hit the portal of the tunnel, and the maximum rebound height at the portal would be about 1.2 m. To prevent such catastrophic failure, active protective net was recommend at each unstable rock zone, and retaining wall with a height of 5 m behind the portal. The results of this study confirm the effectiveness of a UAV photogrammetry-based method for reliable and cost-effective engineering geological surveying of unstable rock mass in complex terrain.