It has been accepted that industry agglomeration has a significant impact on city-level productivity. However, very few researches have investigated the impact of industry agglomeration from the perspective of spatial distribution of firms and mutual effects between firms. This paper uses a network-based, within-city approach to shed new light on the impact of industry agglomeration. Based on a data set that includes records of all Chinese industrial firms from 1998 to 2007, we construct a network of firms for each prefectural level city and characterize within-city industry agglomeration using principal components obtained from network-based measurements and dimension reduction. Those characterizations allow us to examine the impact of industry agglomeration on total factor productivity through panel data regression. We obtain three principal components: modularity, scale, and diversity to characterize city-level industry agglomeration. Panel data regression reveals that all three principal components are significantly, positively related to a city’s total factor productivity. However, as the scale of firms in a city increases, the impact of agglomeration decreases. Meanwhile, relative to the impact of modularity and scale, the impact of diversity on TFP is weaker.