The observation of γ-ray sources above 100 TeV is one of the most intriguing fields in the investigation of the origin of cosmic rays. The YangBaJing Hybrid Array (YBJ-HA) experiment consist of scintillation detectors and underground muon detectors, the main objective is to measure gamma rays >10 TeV. In this study, we investigated an energy estimation method based on particle density and fixed distance from the shower axis by conducting a detailed Monte Carlo simulation of an extensive air shower using the YBJ-HA in Tibet. By comparing the results of the new method with those of other previously used estimators, we found that the new method can effectively improve the energy resolution of gamma rays. At an energy of 100 TeV for a zenith angle θ < 30°, the new energy estimation method has an improvement about 10% compared for the other estimators.