For both the positive economic benefit and the negative ecological impact of the rapid expansion of oil palm plantations in tropical developing countries, it is significant to achieve accurate detection for oil palm trees in large-scale areas. Especially, growing status observation and smart oil palm plantation management enabled by such accurate detections would improve plantation planning, oil palm yield, and reduce manpower and consumption of fertilizer. Although existing studies have already reached a high accuracy in oil palm tree detection, rare attention has been paid to automated observation of each single oil palm tree’s growing status. Nowadays, with its high spatial resolution and low cost, Unmanned Aerial Vehicle (UAV) has become a promising tool for monitoring the growing status of individual oil palms. However, the accuracy is still a challenging issue because of the extreme imbalance and high similarity between different classes. In this paper, we propose a Multi-class Oil PAlm Detection approach (MOPAD) to reap both accurate detection of oil palm trees and accurate monitoring of their growing status. Based on Faster RCNN, MOPAD combines a Refined Pyramid Feature (RPF) module and a hybrid class-balanced loss module to achieve satisfying observation of the growing status for individual oil palms. The former takes advantage of multi-level features to distinguish similar classes and detect small oil palms, and the latter effectively resolves the problem of extremely imbalanced samples. Moreover, we elaborately analyze the distribution of different kinds of oil palms, and propose a practical workflow for detecting oil palm vacancy. We evaluate MOPAD using three large-scale UAV images photographed in two sites in Indonesia (denoted by Site 1 and Site 2), containing 363,877 oil palms of five categories: healthy palms, dead palms, mismanaged palms, smallish palms and yellowish palms. Our proposed MOPAD achieves an F1-score of 87.91% (Site 1) and 99.04% (Site 2) for overall oil palm tree detection, and outperforms other state-of-the-art object detection methods by a remarkable margin of 10.37–17.09% and 8.14%-21.32% with respect to the average F1-score for multi-class oil palm detection in Site 1 and Site 2, respectively. Our method demonstrates excellent potential for individual oil palm tree detection and observation of growing status from UAV images, leading to more precise and efficient management of oil palm plantations.