Grain size is an important agronomic trait that influences the yield and end-use quality of bread wheat (Triticum aestivum L.). We conducted quantitative trait loci (QTL) analysis using recombinant inbred lines derived from a cross of a semi-dwarf modern variety with large grains (Zenkoujikomugi: Zen) and a landrace with small grains (Chinese Spring: CS). Grain size was evaluated as grain length (GL), grain width (GW), grain thickness (GT), and thousand-grain weight (TGW) under two environmental conditions. A major QTL (QGl.obu-4B) associated with GL was detected in both environments near the Rht-B1 locus for the semi-dwarf trait on chromosome 4B, despite Zen and CS having the same Rht-B1a allele. No QTL for GW was found in the QGl.obu-4B region, and QTLs for GT and TGW was detected in only one environment. To validate the effect of QGl.obu-4B, we developed two backcrossed populations in the CS and Zen genetic backgrounds. The Zen-derived allele conferred larger GL, GW, and GT, resulting in greater TGW (13.7% increase compared to the CS-derived allele) in the CS background. In contrast, the QGl.obu-4B effect was smaller in the Zen background, and the Zen-derived allele increased TGW by only 7.0%. The QTL was located within the 168-Mbp region tightly linked to the Rht-B1 locus. Further characterization and fine-mapping of QGl.obu-4B will facilitate the use of this allele by marker-assisted selection in wheat breeding programs.