Recently, progress has been made in the understanding of anomalous vibrational excitations in amorphous solids. In the lowest-frequency region, the vibrational spectrum follows a non-Debye quartic law, which persists up to zero frequency without any frequency gap. This gapless vibrational density of states (vDOS) suggests that glasses are on the verge of instability. This feature of marginal stability is now highlighted as a key concept in the theories of glasses. In particular, the elasticity theory based on marginal stability predicts the gapless vDOS. However, this theory yields a quadratic law and not the quartic law. To address this inconsistency, we presented a new type of instability, which is different from the conventional one, and proposed that amorphous solids are marginally stable considering the new instability in the preceding study [M. Shimada, H. Mizuno and A. Ikeda, Soft Matter, 2020, 16, 7279]. In this study, we further extend and detail the results for these instabilities. By analyzing various examples of disorder, we demonstrate that real glasses in finite spatial dimensions can be marginally stable by the proposed novel instability.