To improve the yield of sweet peppers in Japan, we investigated the relationships between plant growth characteristics and fruit yield, with a particular focus on dry matter production. Fruit fresh weight (i.e., fruit yield) was highly correlated with fruit dry weight. Aboveground dry matter production was correlated with both light use efficiency and intercepted photosynthetically active radiation. Although the number of fruits was correlated with the fruit set ratio, no correlation was found between the number of fruit and fraction of dry matter distributed to fruit. The correlation between the fraction of dry matter partitioned to fruit and fruit dry weight varied by plant stage, potentially due to the yield fluctuation called “flush”. Together, these findings suggest that to improve the yield of sweet peppers in Japan, we should focus first on improving light use efficiency.