Mineralogy differences in clastic sediments have a major impact on the diagenetic pathway and the evolution of pore systems between various lithofacies during burial. There is, however, a lack of studies on shale diagenesis since fine-grained particles are difficult to image and observe directly. Here, a quantitative method by pore counting was performed on the Wufeng-Longmaxi shale from Sichuan Basin, China with siliceous, argillaceous, and calcareous lithofacies, which makes it ideal for studying the impact of diagenesis on pore systems in different lithofacies experiencing the same burial history. The siliceous shale was formed under the anoxic reducing conditions, while argillaceous and calcareous shales were formed under dysoxic-to-oxic conditions. Authigenic quartz, of mostly biogenic origin, accounts for about 60% and 12% of the total quartz content in siliceous and calcareous shales, respectively, whereas it in argillaceous shale is almost detrital origin. The typical diagenetic sequence has the following order: compaction, cementation, clay transformation, dissolution, and organic matter (OM) maturation and its accompanying migrated OM filling. The quantitative calculation reveals that the initial porosity rapidly declines owing to compaction and cementation, while dissolution and OM maturation increase porosity to some extent. Specifically, the dominant factor resulting in porosity reduction varies in different lithofacies, which is quartz cementation and compaction for siliceous and argillaceous shales, respectively, and is both cementation (mainly quartz and carbonates) and compaction for calcareous shales. Additionally, the matrix composition controls the dominant diagenetic events in different lithofacies during the eogenetic (early diagenetic) stage, as indicated by the cementation of authigenic quartz for siliceous and compaction of ductile clays for argillaceous shales. Overall, diagenesis determines the final status of the initial pore space during the mesogenetic (middle diagenetic) stage, which further controls the abundance and distribution of migrated OM, and thus the development of the dominant OM pores in the entire pore spectrum.