Smart control of window behavior is a means of effectively reducing concentrations of indoor PM_2.5 (particulate matter with aerodynamic diameter less than 2.5 μm) in naturally ventilated residential buildings without indoor air cleaning devices. This study aimed to develop a reinforcement learning approach to automatically control window behavior in real time for mitigation of indoor PM_2.5 pollution. The proposed method trains the window controller with the use of a deep Q-network (DQN) in a specific naturally ventilated apartment in the course of a month. The trained controller can then be employed to control window behavior in order to reduce the indoor PM_2.5 concentrations in that apartment. The required input data for the controller are the real-time indoor and outdoor PM_2.5 concentrations with a 1-min resolution, which can easily be obtained with low-cost sensors available on the market. A series of simulations were conducted in a virtual typical apartment in Beijing and a real apartment in Tianjin. The results show that, compared with the baseline I/O ratio algorithm, the proposed reinforcement learning window-control algorithm reduced the average indoor PM_2.5 concentration by 12.80% in a one-year period. Furthermore, the proposed algorithm reduced the indoor PM_2.5 concentrations in the real apartment by 9.11% when compared with the I/O ratio algorithm and by 7.40% when compared with real window behavior.