More than two-thirds of human in the world view rice or wheat as their diet, rice and wheat are grown in some regions of China and other countries in Asian. However, a variety of diseases can affect the growth of rice and wheat, reducing their harvest and even cause famine in some areas. Diseases in leaves, as a kind of diseases, have negative impacts on plants. Under this background, quickly and accurately recognition method is necessary to take in practice and educe the loss. In order to solve this problem, this article aims at three kinds of rice leaf diseases and two kinds of wheat leaf diseases, collects 40 images of each leaf diseases and enhances them. And aims to improve the Visual Geometry Group Network-16(VGG16) model based on the idea of multi-task learning and then use the pre-training model on ImageNET for transfer learning and alternating learning. The accuracy of such model is 97.22% for rice leaf diseases and 98.75% for wheat leaf diseases. Through comparative experiments, it is proved that the effects of this method are better than single-task model, reuse-model method in transfer learning, resnet50 model and densenet121 model. The experimental results show that the improved VGG16 model and multi-task transfer learning method proposed in this article can recognize rice leaf diseases and wheat leaf diseases at the same time, which provides a reliable method for recognizing leaf diseases of many plants.