Skip to main content
SpringerLink
Log in

Textual classification of SEC comment letters

  • Published:
Review of Accounting Studies Aims and scope Submit manuscript

Abstract

This study examines the impact of SEC comment letters on future financial reporting outcomes and earnings credibility. Naïve Bayesian classification identifies comment letters associated with future restatements and write-downs. An investor attention-based quantitative measure of importance, using EDGAR downloads, also predicts these outcomes. Disclosure-event abnormal returns, revenue recognition comments, and the number of letters in a conversation appear to be useful quantitative metrics for classifying importance in certain settings. This study also documents trends in comment letter topics over time and identifies topics associated with the textual and quantitative classifications of importance, providing insights into the factors that draw investor attention and that relate to future restatements and write-downs. Innocuous comment letters are associated with improvements in earnings credibility following comment letter reviews.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Examples of short seller research that uses issues raised in comment letters include presentations by Greenlight Capital on Green Mountain Coffee, Pershing Square on Herbalife, and Prescience Point on Boulder Brands. http://online.wsj.com/public/resources/documents/EinhornGMCRpresentation_Oct2011_VIC.pdf. Retrieved 7 September, 2020.http://factsabout-herbalife.com/wp-content/uploads/2013/01/Who-wants-to-be-a-Millionaire.pdf. Retrieved 1 February 2013.https://www.presciencepoint.com/research/research-archives/boulder-brands-inc-bdbd/. Retrieved 7 September, 2020.

  2. Care needs to be taking cleaning the raw EDGAR log file data set to accurately count comment letter downloads, which are usually filed as PDF documents. Ryans, J., 2017. Using the EDGAR log file data set. Working paper, London Business School.

  3. As described previously, the use of multi-word features such as bigrams, which are used in this study, can allow for some preservation of word order in naïve Bayesian analysis.

  4. See Securities and Exchange Commission. Staff Observations in the Review of Executive Compensation Disclosure. September 10, 2007. http://www.sec.gov/divisions/corpfin/guidance/execcompdisclosure.htm. Accessed 7 September 2020.

References

  • Antweiler, W., & Frank, M. Z. (2004). Is all that talk just noise? The information content of internet stock message boards. Journal of Finance, 59(3), 1259–1294.

    Google Scholar 

  • Ball, R., & Brown, P. (1968). An empirical evaluation of accounting income numbers. Journal of Accounting Research, 6(2), 159–178.

    Google Scholar 

  • Bao, Y., & Datta, A. (2014). Simultaneously discovering and quantifying risk types from textual risk disclosures. Management Science, 60(6), 1371–1391.

    Google Scholar 

  • Bartov, E., Givoly, D., & Hayn, C. (2002). The rewards to meeting or beating earnings expectations. Journal of Accounting and Economics, 33(2), 173–204.

    Google Scholar 

  • Beaver, W. H. (1968). The information content of annual earnings announcements. Journal of Accounting Research, 1968, 67–92.

    Google Scholar 

  • Bens, D. A., Cheng, M., & Neamtiu, M. (2016). The impact of SEC disclosure monitoring on the uncertainty of fair value estimates. The Accounting Review, 91(2), 349–375.

    Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

    Google Scholar 

  • Bloomfield, R. J. (2002). The “incomplete revelation hypothesis” and financial reporting. Accounting Horizons, 16(3), 233–243.

    Google Scholar 

  • Bozanic, Z., Dietrich, J. R., & Johnson, B. A. (2017). SEC comment letters and firm disclosure. Journal of Accounting and Public Policy, 36(5), 337–357.

    Google Scholar 

  • Brown, S. V., Tian, X., & Tucker, J. W. (2018). The spillover effect of SEC comment letters on qualitative corporate disclosure: Evidence from the risk factor disclosure. Contemporary Accounting Research, 35(2), 622–656.

    Google Scholar 

  • Bryan, S. H. (1997). Incremental information content of required disclosures contained in management discussion and analysis. The Accounting Review, 72(2), 285–301.

    Google Scholar 

  • Carhart, M. M. (1997). On persistence in mutual fund performance. Journal of Finance, 52(1), 57–82.

    Google Scholar 

  • Cassell, C. A., Dreher, L. M., & Myers, L. A. (2013). Reviewing the SEC’s review process: 10-K comment letters and the cost of remediation. The Accounting Review, 88(6), 1875–1908.

    Google Scholar 

  • Christensen, H. B., Hail, L., & Leuz, C. (2013). Mandatory IFRS reporting and changes in enforcement. Journal of Accounting and Economics, 56(2–3), 147–177.

    Google Scholar 

  • Collins, D. W., & Kothari, S. P. (1989). An analysis of intertemporal and cross-sectional determinants of earnings response coefficients. Journal of Accounting and Economics, 11(2), 143–181.

    Google Scholar 

  • Cunningham, L. M., Johnson, B. A., Johnson, E. S., & Lisic, L. L. (2019). The switch up: An examination of changes in earnings management after receiving sec comment letters. Contemporary Accounting Research, 37(2), 917–944.

    Google Scholar 

  • Dagan, I., Feldman, R., Hirsh, H., (1996). Keyword-based browsing and analysis of large document sets, Proceedings of the Fifth Annual Symposium on Document Analysis and Information Retrieval (pp. 191–208). University of Nevada.

  • Davis, A. K., Piger, J. M., & Sedor, L. M. (2012). Beyond the numbers: Measuring the information content of earnings press release language. Contemporary Accounting Research, 29(3), 845–868.

    Google Scholar 

  • De Franco, G., Vasvari, F. P., Vyas, D., & Wittenberg-Moerman, R. (2013). Debt analysts’ views of debt-equity conflicts of interest. The Accounting Review, 89(2), 571–604.

    Google Scholar 

  • Dechow, P. M., Ge, W., Larson, C. R., & Sloan, R. G. (2011). Predicting material accounting misstatements. Contemporary Accounting Research, 28(1), 17–82.

    Google Scholar 

  • Dechow, P. M., Huson, M. R., & Sloan, R. G. (1994). The effect of restructuring charges on executives’ cash compensation. The Accounting Review, 69(1), 138–156.

    Google Scholar 

  • Dechow, P. M., Lawrence, A., & Ryans, J. (2016). SEC comment letters and insider sales. The Accounting Review, 91(2), 401–439.

    Google Scholar 

  • Dechow, P. M., Sloan, R. G., & Sweeney, A. P. (1995). Detecting earnings management. The Accounting Review, 70(2), 193–225.

    Google Scholar 

  • DeFond, M. L., & Jiambalvo, J. (1991). Incidence and circumstances of accounting errors. The Accounting Review, 66(3), 643–655.

    Google Scholar 

  • Drake, M., Johnson, B., Roulstone, D., & Thornock, J. (2019). Is there information content in information acquisition. The Accounting Review, 95(2), 113–139.

    Google Scholar 

  • Drake, M. S., Roulstone, D. T., & Thornock, J. R. (2015). The determinants and consequences of information acquisition via EDGAR. Contemporary Accounting Research, 32(3), 1128–1161.

    Google Scholar 

  • Duro, M., Heese, J., & Ormazabal, G. (2019). The effect of enforcement transparency: Evidence from SEC comment-letter reviews. Review of Accounting Studies, 24(3), 780–823.

    Google Scholar 

  • Dyck, A., Morse, A., & Zingales, L. (2010). Who blows the whistle on corporate fraud? Journal of Finance, 65(6), 2213–2253.

    Google Scholar 

  • Dye, R. A. (1985). Disclosure of nonproprietary information. Journal of Accounting Research, 23(1), 123–145.

    Google Scholar 

  • Dyer, T., Lang, M., & Stice-Lawrence, L. (2017). The evolution of 10-K textual disclosure: Evidence from latent Dirichlet allocation. Journal of Accounting and Economics, 64(2–3), 221–245.

    Google Scholar 

  • Engelberg, J. E., & Parsons, C. A. (2011). The causal impact of media in financial markets. Journal of Finance, 66(1), 67–97.

    Google Scholar 

  • Feldman, R., Dagan, I., (1995). Knowledge discovery in textual databases KDT. Proceedings of the First International Conference on Knowledge Discovery and Data Mining (pp. 112–117). Association for Computing Machinery.

  • Feldman, R., Govindaraj, S., Livnat, J., & Segal, B. (2010). Management’s tone change, post earnings announcement drift and accruals. Review of Accounting Studies, 15(4), 915–953.

    Google Scholar 

  • Feroz, E.H., Park, K., Pastena, V.S., (1991). The financial and market effects of the SEC’s accounting and auditing enforcement releases. Journal of Accounting Research 29(Supp.), 107–142.

  • Francis, J., LaFond, R., Olsson, P., & Schipper, K. (2005). The market pricing of accruals quality. Journal of Accounting and Economics, 39(2), 295–327.

    Google Scholar 

  • Francis, J., Nanda, D., & Olsson, P. (2008). Voluntary disclosure, earnings quality, and cost of capital. Journal of Accounting Research, 46(1), 53–99.

    Google Scholar 

  • Francis, J. R. (2011). A framework for understanding and researching audit quality. Auditing: A Journal of Practice & Theory, 30(2), 125–152.

    Google Scholar 

  • Gietzmann, M. B., & Pettinicchio, A. K. (2013). External auditor reassessment of client business risk following the issuance of a comment letter by the SEC. European Accounting Review, 23(1), 57–85.

    Google Scholar 

  • Hayn, C. (1995). The information content of losses. Journal of Accounting and Economics, 20(2), 125–153.

    Google Scholar 

  • Heese, J., Khan, M., & Ramanna, K. (2017). Is the SEC captured? Evidence from comment-letter reviews. Journal of Accounting and Economics, 64(1), 98–122.

    Google Scholar 

  • Hennes, K. M., Leone, A. J., & Miller, B. P. (2014). Determinants and market consequences of auditor dismissals after accounting restatements. The Accounting Review, 89(3), 1051–1082.

    Google Scholar 

  • Hirshleifer, D., & Teoh, S. H. (2003). Limited attention, information disclosure, and financial reporting. Journal of Accounting and Economics, 36(1), 337–386.

    Google Scholar 

  • Hoberg, G., & Lewis, C. (2017). Do fraudulent firms produce abnormal disclosure? Journal of Corporate Finance, 43(1), 58–85.

    Google Scholar 

  • Holthausen, R. W., & Verrecchia, R. E. (1988). The effect of sequential information releases on the variance of price changes in an intertemporal multi-asset market. Journal of Accounting Research, 26(1), 82–106.

    Google Scholar 

  • Hribar, P., & Jenkins, N. T. (2004). The effect of accounting restatements on earnings revisions and the estimated cost of capital. Review of Accounting Studies, 9(2–3), 337–356.

    Google Scholar 

  • Hribar, P., Kravet, T., & Wilson, R. (2014). A new measure of accounting quality. Review of Accounting Studies, 19(1), 506–538.

    Google Scholar 

  • Huang, A. H., Lehavy, R., Zang, A. Y., & Zheng, R. (2018). Analyst information discovery and interpretation roles: A topic modeling approach. Management Science, 64(6), 2833–2855.

    Google Scholar 

  • Huang, A. H., Zang, A. Y., & Zheng, R. (2014). Evidence on the information content of text in analyst reports. The Accounting Review, 89(6), 2151–2180.

    Google Scholar 

  • Johnston, R., & Petacchi, R. (2017). Regulatory oversight of financial reporting: Securities and exchange commission comment letters. Contemporary Accounting Research, 34(2), 1128–1155.

    Google Scholar 

  • Jung, W. O., & Kwon, Y. K. (1988). Disclosure when the market is unsure of information endowment of managers. Journal of Accounting Research, 26(1), 146–153.

    Google Scholar 

  • Karlgren, J., Cutting, D., (1994). Recognizing text genres with simple metrics using discriminant analysis. Proceedings of the 15th conference on Computational Linguistics (pp. 1071–1075). Association for Computational Linguistics.

  • Kessler, B., Numberg, G., Schutze, H., (1997). Automatic detection of text genre, Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics (pp. 32–38).Association for Computational Linguistics.

  • Kinney, W. R., & McDaniel, L. S. (1989). Characteristics of firms correcting previously reported quarterly earnings. Journal of Accounting and Economics, 11(1), 71–93.

    Google Scholar 

  • Kinney, W. R., Palmrose, Z. V., & Scholz, S. (2004). Auditor independence, non-audit services, and restatements: Was the US government right? Journal of Accounting Research, 42(3), 561–588.

    Google Scholar 

  • Kormendi, R., & Lipe, R. (1987). Earnings innovations, earnings persistence, and stock returns. Journal of Business, 60(3), 323–345.

    Google Scholar 

  • Kothari, S. P., Li, X., & Short, J. E. (2009a). The effect of disclosures by management, analysts, and business press on cost of capital, return volatility, and analyst forecasts: A study using content analysis. The Accounting Review, 84(5), 1639–1670.

    Google Scholar 

  • Kothari, S. P., Shu, S., & Wysocki, P. D. (2009b). Do managers withhold bad news? Journal of Accounting Research, 47(1), 241–276.

    Google Scholar 

  • La Porta, R., Lopez-de Silanes, F., & Shleifer, A. (2006). What works in securities laws? Journal of Finance, 61(1), 1–32.

    Google Scholar 

  • Lang, M., & Lundholm, R. (1993). Cross-sectional determinants of analyst ratings of corporate disclosures. Journal of Accounting Research, 31(2), 246–271.

    Google Scholar 

  • Larcker, D. F., & Zakolyukina, A. A. (2012). Detecting deceptive discussions in conference calls. Journal of Accounting Research, 50(2), 495–540.

    Google Scholar 

  • Laurion, H., Lawrence, A., & Ryans, J. (2017). U.S. audit partner rotation. The Accounting Review, 92(3), 209–237.

    Google Scholar 

  • Law, K. K. F., & Mills, L. F. (2015). Taxes and financial constraints: Evidence from linguistic cues. Journal of Accounting Research, 53(4), 777–819.

    Google Scholar 

  • Lawrence, A., Sloan, R., & Sun, Y. (2013). Non-discretionary conservatism: Evidence and implications. Journal of Accounting and Economics, 56(2), 112–133.

    Google Scholar 

  • Leuz, C., & Wysocki, P. (2016). The economics of disclosure and financial reporting regulation: Evidence and suggestions for future research. Journal of Accounting Research, 54(2), 525–622.

    Google Scholar 

  • Lewis, D.D., (1998). Naïve (Bayes) at forty: The independence assumption in information retrieval. Machine learning: ECML-98 (pp. 4–15). Springer.

  • Li, F. (2008). Annual report readability, current earnings, and earnings persistence. Journal of Accounting and Economics, 45(2), 221–247.

    Google Scholar 

  • Li, F. (2010). The information content of forward-looking statements in corporate filings. A naïve Bayesian machine learning approach. Journal of Accounting Research, 48(5), 1049–1102.

    Google Scholar 

  • Liu, L. L., Raghunandan, K., & Rama, D. (2009). Financial restatements and shareholder ratifications of the auditor. Auditing: A Journal of Practice & Theory, 28(1), 225–240.

    Google Scholar 

  • Livnat, J., & Mendenhall, R. R. (2006). Comparing the post–earnings announcement drift for surprises calculated from analyst and time series forecasts. Journal of Accounting Research, 44(1), 177–205.

    Google Scholar 

  • Ljungqvist, A., & Qian, W. (2016). How constraining are limits to arbitrage? Evidence from a recent financial innovation. Review of Financial Studies, 29(8), 1975–2028.

    Google Scholar 

  • Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. Journal of Finance, 66(1), 35–65.

    Google Scholar 

  • Loughran, T., & McDonald, B. (2016). Textual analysis in accounting and finance: A survey. Journal of Accounting Research, 54(4), 1187–1230.

    Google Scholar 

  • Loughran, T., McDonald, B., (2017). The use of EDGAR filings by investors. Journal of Behavioral Finance 18(2), 231–248).

  • Mendenhall, R.R., Nichols, W.D., (1988). Bad news and differential market reactions to announcements of earlier-quarters versus fourth-quarter earnings. Journal of Accounting Research 26(Supp.), 63–86.

  • Mosteller, F., & Wallace, D. L. (1984). Applied Bayesian and classical inference. New York: Springer-Verlag.

    Google Scholar 

  • Naughton, J. P., Rogo, R., Sunder, J., & Zhang, R. (2018). SEC monitoring of foreign firms’ disclosures. Review of Accounting Studies, 23(4), 1355–1388.

    Google Scholar 

  • Palmrose, Z. V., Richardson, V. J., & Scholz, S. (2004). Determinants of market reactions to restatement announcements. Journal of Accounting and Economics, 37(1), 59–89.

    Google Scholar 

  • Pang, B., Lee, L., Vaithyanathan, S., (2002). Thumbs up?: Sentiment classification using machine learning techniques, Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing . (pp. 79–86). Association for Computational Linguistics.

  • Peterson, K. (2012). Accounting complexity, misreporting, and the consequences of misreporting. Review of Accounting Studies, 17(1), 72–95.

    Google Scholar 

  • Quinn, K. M., Monroe, B. L., Colaresi, M., Crespin, M. H., & Radev, D. R. (2010). How to analyze political attention with minimal assumptions and costs. American Journal of Political Science, 54(1), 209–228.

    Google Scholar 

  • Ramanna, K., & Watts, R. L. (2012). Evidence on the use of unverifiable estimates in required goodwill impairment. Review of Accounting Studies, 17(4), 749–780.

    Google Scholar 

  • Robinson, J. R., Xue, Y., & Yu, Y. (2011). Determinants of disclosure noncompliance and the effect of the SEC review: Evidence from the 2006 mandated compensation disclosure regulations. The Accounting Review, 86(4), 1415–1444.

    Google Scholar 

  • Sandler, L., (2013). Muddy waters secret China weapon is on SEC website. Bloomberg News. https://www.bloomberg.com/news/articles/2013-02-19/muddy-waters-secret-china-weapon-is-on-sec-website. Accessed 7 September 2020.

  • SEC, (2015). FY 2016 congressional budget justification, FY 2016 annual performance plan, and FY 2014 annual performance report. U.S. Securities and Exchange Commission. http://www.sec.gov/about/reports/secfy16congbudgjust.shtml. Accessed 7 September 2020.

  • Talley, E., & O’Kane, D. (2012). The measure of a MAC: A machine-learning protocol for analyzing force majeure clauses in ma agreements. Journal of Institutional and Theoretical Economics, 168(1), 181–201.

    Google Scholar 

  • Teoh, S. H., & Wong, T. (1993). Perceived auditor quality and the earnings response coefficient. The Accounting Review, 68(2), 346–366.

    Google Scholar 

  • Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media in the stock market. Journal of Finance, 62(3), 1139–1168.

    Google Scholar 

  • You, H., & Zhang, X. J. (2009). Financial reporting complexity and investor underreaction to 10-K information. Review of Accounting Studies, 14(4), 559–586.

    Google Scholar 

Download references

Acknowledgments

This paper is based on my dissertation from the University of California at Berkeley. I especially thank my dissertation committee members: Patricia Dechow (chair), Alastair Lawrence, Panos Patatoukas, Richard Sloan, and Stephen Davidoff Solomon. I have received valuable comments and suggestions from Russell Lundholm (editor), two anonymous reviewers, John Barrios, Robert Bartlett, Stefano DellaVigna, Paul Fischer, Miles Gietzmann, Mark Huson, Greg Miller, Lillian Mills, Miguel Minutti-Meza, Reining Petacchi, Gordon Phillips, Lakshmanan Shivakumar, Shyam Sunder, Phil Stocken, İrem Tuna, Anastasia Zakolyukina, Luigi Zingales, and workshop participants at Cornell University, Dartmouth College, IESE, London Business School, the University of California at Los Angeles, the University of Texas at Austin, the University of Toronto, Yale University, the 2015 FARS Conference, 2015 EAA Annual Congress, 2015 PCAOB Conference, and the 2018 Carnegie Mellon University Accounting Mini-Conference.

Author information

Authors and Affiliations

  1. London Business School, London, NW1 4SA, UK

    James P. Ryans

Authors
  1. James P. Ryans
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James P. Ryans.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1. Naïve Bayes classification terms over time

Table 11 Naïve Bayes classification terms over time
Full size table

Appendix 2. Variable definitions

Table 12 Variable definitions
Full size table

Appendix 3. Supplemental latent Dirichlet allocation analysis

Table 13 Top terms associated with comment letter topics
Full size table
figure 4
figure 5

These figures present examples of the variation in standard letter language from a pre- and post-2010 comment letter, which the latent Dirichlet allocation topic analysis identifies within the letters as two different topics. Depending on the scope of the review and if there are requests for revisions, these sections of the comment letters will exhibit additional variation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ryans, J.P. Textual classification of SEC comment letters. Rev Account Stud 26, 37–80 (2021). https://doi.org/10.1007/s11142-020-09565-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11142-020-09565-6

Keywords

JEL

Search

Navigation