The role of brain regions in the relationship between psychological stress and sleep quality is unclear. This study investigates the neuroanatomical basis of the association between psychological stress and sleep quality. Data were collected using the Pittsburgh Sleep Quality Index, the Psychosomatic Tension Relaxation Inventory, and voxel-based morphometry among 318 healthy students. The results showed that psychological stress was negatively correlated with sleep quality. According to the mediation analysis results, the correlation between psychological stress and sleep quality was partially mediated by the region of the bilateral inferior temporal gyrus. These findings suggest that there is a strong link between sleep quality and psychological stress, highlighting the gray matter volume of the bilateral inferior temporal gyrus related to emotional processing, which plays an essential role in improving sleep quality.