Skip to main content

Advertisement

SpringerLink
Log in

Evaluating a Natural Language Processing Approach to Estimating KSA and Interest Job Analysis Ratings

  • Original Paper
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Collecting job analysis ratings for a large number of jobs via surveys, interviews, or focus groups can put a very large burden on organizations. In this study, we describe and evaluate a streamlined, natural language processing-based approach to estimating (a) the importance of various knowledges, skills, abilities, and other characteristics (KSAOs) to jobs, and (b) how descriptive various interests are of work on a job. Specifically, we evaluate whether we can train a machine to accurately estimate KSAO ratings for jobs using job description and task statement text as the sole input. Data for 963 occupations from the U.S. Department of Labor’s Occupational Information Network (O*NET) system and an independent set of 229 occupations from a large organization provided the basis for the evaluation. Our approach produced KSAO predictions that had cross-validated correlations with subject matter expert (SME) ratings of knowledges, skills, abilities, and interests of .74, .80, .75, and .84, respectively (on average, across the 126 KSAOs examined). We found clear evidence for the validity of machine-based predictions based on (a) convergence among machine-based and SME-furnished ratings, (b) conceptually meaningful patterns of prediction model regression coefficients among the KSAOs examined, and (c) conceptual relevance of top predictor models underlying related clusters of KSAOs. We also found that prediction models developed on O*NET data produced meaningful results when applied to an independent set of job descriptions and tasks. Implications of this work, as well as suggested directions for future job analysis research and practice, are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Although not of historical focus in the I-O psychology job analysis literature, interests play a critical role in occupational analysis in organizational practice and applied vocational, organizational, and educational psychology (Su, 2020). Vocational interests have long been a central component of vocational counseling and career exploration (Brown & Lent, 2013). Additionally, recent research has clearly linked vocational interests on theoretical and empirical grounds to job performance (e.g., Nye, Su, Rounds, & Drasgow, 2017; Schmidt, 2014; Su, 2020; Van Iddekinge, et al., 2011). As such, we deemed interests as important “other characteristics” to examine, given our focus on enterprise-wide job analysis efforts where differentiating among a diverse set of jobs is a central concern.

  2. The Federal occupation to O*NET crosswalk used in this study was developed as part of earlier research to inform a Federal career exploration system, which in turn was based on crosswalk developed by OPM between Federal occupation series codes and the Standard Occupational Classification (SOC) (Russell, 2011). The aforementioned crosswalk was created through a multi-step human subject matter expert (SME) judgment process detailed in Russell (2011). The crosswalk focused on linking Federal and O*NET occupations that were as much a direct match as possible in terms of job duties performed. This does not mean that O*NET occupations not crosswalked to a given Federal occupation were not related to that occupation (e.g., some uncrosswalked occupations may have similar KSAO requirements), simply that they were deemed an insufficient match in term of job duties performed relative to the O*NET occupations ultimately crosswalked to them.

  3. In the O*NET 22.0 database, 1,100 occupations had descriptions, 966 had KSA ratings, 974 had interest ratings, and 963 had core tasks. The subset of occupations examined here provides nearly full coverage of the 974 occupations that constitute occupations covered by O*NET’s data collection plan (https://www.onetcenter.org/taxonomy/2010/data_coll.html).

  4. These sources can be found online at the following URLs: PCS: https://www.opm.gov/policy-data-oversight/classification-qualifications/classifying-general-schedule-positions/#url=Standards; JGS: https://www.opm.gov/policy-data-oversight/classification-qualifications/classifying-federal-wage-system-positions/#url=Standards), and HOGF: https://www.opm.gov/policy-data-oversight/classification-qualifications/classifying-general-schedule-positions/occupationalhandbook.pdf.

  5. Given the large drop in words, we also examined models using a 1% threshold for eliminating words and found no substantive difference in model cross-validation results (e.g., cross-validated r’s did not differ by more than .01). This was expected given words with such a low base rate would not be expected to offer much predictive value upon cross-validation. We also attempted to examine models with even lower thresholds for eliminating words (0.75%, and 0.5%) but were not able to fit models because as the sample was split for modeling fitting purposes, several words wound up having no variance across occupations.

  6. Interested readers can find more details on how components factor into the estimation of SPLS regression coefficients in the SPLS algorithm section of Chun and Keles (2010).

  7. We created an Open Science Framework respository for this article at https://osf.io/jbtkp/. This repository  includes R code for (a) initial processing of the O*NET data, (b) cleaning of O*NET occupation text in preparation for modeling, (c) training of SPLS models and application to O*NET occupation test sets, and (d) consolidation of results across KSAOs. In addition to code, we provide key data files we started with and that are produced by running the code referenced above.

  8. Follow-up analyses revealed setting these bounds had no substantive impact on conclusions drawn in this study. We imposed these bounds to allow for more direct comparison to results in O*NET and to permit future researchers to derive predictions that were bounded by values used the O*NET ratings scales for KSAOs.

  9. To facilitate quick, visual identification of the top language-based predictors for each KSAO, we have also provided supplemental materials that show word clouds of the top 50 positive predictors in the final SPLS KSAO models (see Supplement C).

  10. We thank an anonymous reviewer for highlighting this positive finding.

  11. When interpreting these results, note that residuals will be larger for interests, as they are based on a 7-point rating scale, whereas the KSAs are based on a 5-point rating scale.

  12. To be clear, the judgment of “conceptual meaningfulness” here is somewhat subjective. It is based on a review of these results by a subset of the authors of this manuscript, all of whom have extensive research and applied experience in the areas of individual differences and occupational analysis. Thus, the perspective offered here is based on decades of applied research experience.

  13. We thank an anonymous reviewer for suggesting we conduct further analyses to explain the relatively high profile correlations among uncrosswalked occupations.

  14. In the context of text analysis, a “dictionary” is simply a set of words that have been identified as useful for measuring or predicting some construct or concept of interest.

References

  • Amorim, E., Cancado, M., & Veloso, A. (2018). Automated essay scoring in the presence of biased ratings. Proceedings of NAACL-HLT 2018, 229–337. Association for Computational Linguistics.

  • Baranowski, L. E., & Anderson, L. E. (2005). Examining rating source variation in work behavior to KSA linkages. Personnel Psychology, 58(4), 1041–1054.

    Article  Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

    Google Scholar 

  • Bobko, P., Roth, P. L., & Buster, M. L. (2008). A systematic approach for assessing the currency (“up-to-dateness”) of job-analytic information. Public Personnel Management, 37, 261–277.

    Article  Google Scholar 

  • Brannick, M. T., Levine, E. L., & Morgeson, F. P. (2007). Job and work analysis: Methods research, and applications for human resource management (2nd ed.). Sage.

  • Brannick, M. T., Pearlman, K., & Sanchez, J. I. (2017). Work analysis. In J. L. Farr & N. T. Tippins (Eds.), Handbook of Employee Selection (2nd ed., pp. 134–161). Routledge.

    Chapter  Google Scholar 

  • Brown, S. D., & Lent, R. W. (2013). Career development and counseling: Putting theory and research to work, 2nd edition. Wiley.

  • Campion, M. C., Campion, M. A., Campion, E. D., & Reider, M. H. (2016). Initial investigation into computer scoring of candidate essays for personnel selection. Journal of Applied Psychology, 101, 958–975.

    Article  PubMed  Google Scholar 

  • Carter, G. W., Cook, K. W., & Dorsey, D. W. (2009). Career paths: Charting courses to success. Wiley-Blackwell.

  • Carter, G. W., Dorsey, D. W., & Niehaus, J. W. (2004, April). The use of transactional data in occupational analysis: Text-mining of on-line job listings. In J. M. Ford (Chair), Automated text analysis in I/O psychology: Research to practice. Symposium conducted at the Annual Conference of the Society for Industrial and Organizational Psychology, Chicago.

  • Casner-Lotto, J., & Barrington, L. (2006). Are they really ready to work? Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century workforce. Partnership for 21st Century Skills.

  • Cawley, G. C., & Talbot, N. L. C. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11, 2079–2107.

    Google Scholar 

  • Chun, H., & Keles, S. (2010). Sparse partial least squares regression for simultaneous dimension reduction and variable selection. Journal of the Royal Statistical Society: Series B, 72, 3–25.

    Article  Google Scholar 

  • Chung, D, Chun, H., & Keles, S. (2013). spls: Sparse partial least squares (SPLS) regression and classification. (Version 2.2–1). Available from: https://CRAN.R-project.org/package=spls

  • Costanza, D. P., & Fleishman, E. A. (1992). Fleishman Job Analysis Survey (Part III). Management Research Institute.

  • Dawis, R. V., & Lofquist, L. H. (1984). A psychological theory of work adjustment. University of Minnesota Press.

    Google Scholar 

  • Department of the Army (2019). Army Research Institute for the Behavioral and Social Sciences – Request for White Papers: Data Science for Enhancing Job Design (Solicitation Number: W911NF-18-S-0005-FY1906)

  • Dierdorff, E. C., & Norton, J. J. (2011). Summary of procedures for O*NET task updating and new task generation. National Center for O*NET Development. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/TaskUpdating.pdf

  • Efron, B., & Hastie, T. (2016). Computer age statistical inference: Algorithms, evidence, and data science. Cambridge University Press.

    Book  Google Scholar 

  • Feinerer, I., & Hornik, K. (2017). tm: Text mining package. (Version 0.7–1). Available from: https://CRAN.R-project.org/package=tm

  • Fleisher, M. S., & Tsacoumis, S. (2012a). O*NET analyst occupational skills ratings: Procedures update. Human Resources Research Organization. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/AOSkills_ProcUpdate.pdf

  • Fleisher, M. S., & Tsacoumis, S. (2012b). O*NET analyst occupational ability ratings: Procedures update. Human Resources Research Organization. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/AnalystProcUpdate.pdf

  • Fleishman, E. A., Constanza, D. P., Marshall-Mies, J., Wetrogan, L. I., & Uhlman, C. E. (1995a). Knowledges. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jenneret, & E. A. Fleishman (Eds.), Development of prototype Occupational Information Network (O*NET) content model (pp. 4–1 - 4–23). Utah Department of Employment Security.

  • Fleishman, E. A., & Mumford, M. D. (1991). Evaluating classifications of job behavior: A construct validation of the ability requirement scales. Personnel Psychology, 44, 523–575.

    Article  Google Scholar 

  • Fleishman, E. A., & Reilly, M. E. (1992). Handbook of human abilities: Definitions, measurements, and job task requirements. Consulting Psychologists Press.

    Google Scholar 

  • Fleishman, E. A., Wetrogan, L. I., Uhlman, C. E., & Marshall-Mies, J. (1995b). Abilities. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jenneret, & E. A. Fleishman (Eds.), Development of prototype Occupational Information Network (O*NET) content model (pp. 10–1–10–39). Utah Department of Employment Security

  • Frey, C. B., & Osborne, M. A. (2017). The future of employment: How susceptible are jobs to computerization? Technological Forecasting and Social Change, 114, 254–280.

    Article  Google Scholar 

  • Friedman, L., & Harvey, R. J. (1986). Can raters with reduced job descriptive information provide accurate Position Analysis Questionnaire (PAQ) ratings? Personnel Psychology, 39, 779–789.

    Article  Google Scholar 

  • Gael, S. (1988). Job descriptions. In. S. Gael (Ed.), The job analysis handbook for business, industry, and government, Volume 1 (pp. 71–89). Wiley.

  • Gruder, E. J. (2012). Identifying appropriate sources of work information. In M. A. Wilson, W. Bennett, S. G. Gibson, & G. M. Alliger (Eds.), The handbook of work analysis: Methods, systems, applications and science of work measurement in organizations (pp. 31–40). Routledge/Taylor & Francis Group.

    Google Scholar 

  • Harvey, R. J., & Lozada-Larsen, S. R. (1988). Influence of amount of job descriptive information on job analysis rating accuracy. Journal of Applied Psychology, 73, 457–461.

    Article  Google Scholar 

  • Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.

  • Holland, J. L. (1997). Making vocational choices: A theory of vocational personalities and work environments (3rd ed.). Psychological Assessment Resources.

  • Howard, A. (Ed.) (1995). The changing nature of work. Jossey-Bass.

  • Kern, M. L., Park, G., Eischstaedt, J. C., Schwartz, H. A., Sap, M., Smith, L. K., & Ungar, L. H. (2016). Gaining insights from social media language: Methodologies and challenges. Psychological Methods, 21, 507–525.

    Article  PubMed  Google Scholar 

  • Kjell, O. N. E., Sikström, S., Kjell, K., & Schwartz, H. A. (2022). Natural language analyzed with AI-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy. Scientific Reports, 12, 3918.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. PNAS Proceedings of the National Academy of Sciences of the United States of America, 110, 5802–5805.

    Article  PubMed  Google Scholar 

  • Kosinski, M., Wang, Y., Lakkaraju, H., & Leskovec, J. (2016). Mining big data to extract patterns and predict real-life outcomes. Psychological Methods, 21, 493–506.

    Article  PubMed  Google Scholar 

  • Kuhn, M., & Johnson, K. (2013). Applied predictive modeling. Springer.

    Book  Google Scholar 

  • Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9, 202–220.

    Article  Google Scholar 

  • Landers, R. N. (2017, April). A crash course in natural language processing. The Industrial-Organizational Psychologist, http://www.siop.org/tip/april17/crash.aspx

  • Landers, R. N., Brusso, R. C., Cavanaugh, K. J., & Collmus, A. B. (2016). A primer on theory-driven web scraping: Automatic extraction of big data from the Internet for use in psychological research. Psychological Methods, 21, 475–492.

    Article  PubMed  Google Scholar 

  • Lievens, F., Sanchez, J. I., & De Corte, W. (2004). Easing the inferential leap in competency modeling: The effects of task-related information and subject matter expertise. Personnel Psychology, 57, 881–904.

    Article  Google Scholar 

  • Liu, B. (2015). Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge University Press.

    Book  Google Scholar 

  • Manning, C. D., & Shutze, H. (1999). Foundations of statistical natural language processing. MIT Press.

    Google Scholar 

  • Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClonsky, D. (2014). The Stanford CoreNLP natural language processing toolkit. Proceeding of the 52nd Annual Meeting of the Association of Computational Linguistic: System Demonstrations (pp. 55–60). Baltimore, MD.

  • McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.

    Article  Google Scholar 

  • McKenny, A. F., Aguinis, H., Short, J. C., & Anglin, A. H. (2018). What doesn’t get measured does exist: Improving the accuracy of computer-aided text analysis. Journal of Management, 44, 2909–2933.

    Article  Google Scholar 

  • McKenny, A. F., Short, J. C., & Payne, G. T. (2013). Using computer-aided text analysis to elevate constructs: An illustration using psychological capital. Organizational Research Methods, 16, 152–184.

    Article  Google Scholar 

  • Michalke, M. (2017a). Using the koRpus package for text analysis. Retrieved July 29, 2017a, from https://cran.r-project.org/web/packages/koRpus/vignettes/koRpus_vignette.pdf

  • Michalke, M. (2017b). koRpus: An R package for text analysis. (Version 0.10–2). Available from: https://reaktanz.de/?c=hacking&s=koRpus

  • Mikolov, T., Sutskever, I, Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems. Retrieved July 8, 2017, from https://arxiv.org/abs/1310.4546

  • Morgeson, F. P., & Campion, M. A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82, 627–655.

    Article  Google Scholar 

  • Morgeson, F. P., & Campion, M. A. (2000). Accuracy in job analysis: Toward an inference-based model. Journal of Organizational Behavior, 21, 819–827.

    Article  Google Scholar 

  • Mumford, M. D., & Peterson, N. G. (1995). Skills. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jenneret, & E. A. Fleishman (Eds.), Development of prototype Occupational Information Network (O*NET) content model (pp. 3–1 - 3–75). Utah Department of Employment Security.

  • National Center for O*NET Development. (n.d.) Content model. O*NET Resource Center. Retrieved July 8, 2017, from https://www.onetcenter.org/content.html

  • Nye, C. D., Su, R., Rounds, J., & Drasgow, F. (2017). The relationship between interests and performance: An updated meta-analysis. Journal of Vocational Behavior, 98, 138–151.

    Article  Google Scholar 

  • O’Connor, B. P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, and Computers, 32, 396–402.

    Article  Google Scholar 

  • O’Neil, C. (2016). Weapons of math destruction. Crown Publishing Group.

    Google Scholar 

  • Pan, Y., Peng, Y., Hu, T., & Jiebo, L. (2017). Understanding what affects career progression using LinkedIn and Twitter Data. Special Session on Intelligent Data Mining. IEEE Big Data Conference, Boston, MA.

  • Park, G., Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Kosinski, M., Stillwell, D. J., Ungar, L. H., & Seligman, M. E. (2015). Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108, 934–952.

    Article  PubMed  Google Scholar 

  • Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. University of Texas at Austin.

    Google Scholar 

  • Praama, B. Y., & Samo, R. (2015). Personality classification based on Twitter text using Naïve Bayes, KNN, and SVM. 2015 International Conference on Data and Software Engineering (ICoDSE). Yoayakara, Indonesia.

  • Preotiuc-Pietro, D., Carpenter, J., Giorgi, S., & Ungar, L. (2016). Studying the Dark Triad of personality through Twitter behavior. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (CIKM '16). New York.

  • Primoff, E. S. (1975). How to prepare and conduct job element examinations (U.S. Civil Service Commission Technical Study 75–1). Government Printing Office.

  • Putka, D. J., Beatty, A., & Reeder, M. (2018). Modern prediction methods: New perspectives on a common problem. Organizational Research Methods, 21, 689–732.

    Article  Google Scholar 

  • Raymark, P. H., Schmit, M. J., & Guion, R. M. (1997). Identifying potentially useful personality constructs for employee selection. Personnel Psychology, 50, 723–736.

    Article  Google Scholar 

  • Reeder, M. C., & Tsacoumis, S. (2017a). O*NET analyst ratings of occupational skills: Analysis cycle 17 results (2017a No. 0003). Human Resources Research Organization. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/AOSkills_17.pdf

  • Reeder, M. C., & Tsacoumis, S. (2017b). O*NET analyst ratings of occupational abilities: Analysis cycle 17 results (2017b No. 0002). Human Resources Research Organization. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/Wave17.pdf

  • Riloff, E. (1995). Little words can make a big difference for text classification. Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 130–136). Seattle, WA.

  • Rinker, T. W. (2013). qdap: Quantitative discourse analysis package. (Version 2.2.5). University at Buffalo. Available from: http://github.com/trinker/qdap

  • Robinson-Morral, E. J., Hendrickson, C., Gilbert, S., Myers, T., Simpson, K., & Loignon, A. C. (2018). Practical considerations for conducting job analysis linkage exercises. Journal of Personnel Psychology, 17, 12–21.

    Article  Google Scholar 

  • Rounds, J., Armstrong, P. I., Liao, H., Lewis, P., & Rivkin, D. (2008). Second generation occupational interest profiles for the O*NET system: Summary. National Center for O*NET Development. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/SecondOIP_Summary.pdf

  • Rounds, J., Smith, T., Hubert, L., Lewis, P., & Rivkin, D. (1999). Development of Occupational Interest Profiles (OIPs) for O*NET. National Center for O*NET Development. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/OIP.pdf

  • Russell, T. L. (2011). Linking Federal occupational series to the O*NET-SOC 2010 classification. In D. J. Putka & R. A. McCloy (Eds.), Building an interactive federal career exploration and discovery system: Phase I (pp. F11-58). Human Resources Research Organization.

    Google Scholar 

  • Sanchez, J. I., & Levine, E. L. (2000). Accuracy or consequential validity: Which is the better standard for job analysis data? Journal of Organizational Behavior, 21, 809–818.

    Article  Google Scholar 

  • Schmid, H. (1994). Probabilistic part-of-speech tagging using decision trees. Proceedings of International Conference on New Methods in Language Processing, Manchester, UK.

  • Schmidt, F. L. (2014). A general theoretical integrative model of individual differences in interests, abilities, personality traits and academic and occupational achievement: A commentary on four recent articles. Perspectives on Psychological Science, 9, 211–218.

    Article  PubMed  Google Scholar 

  • Short, J. C., Broberg, J. C., Cogliser, C. C., & Brigham, K. H. (2010). Construct validation using computer-aided text analysis (CATA) an illustration using entrepreneurial orientation. Organizational Research Methods, 13, 320–347.

    Article  Google Scholar 

  • Su, R. (2020). The three faces of interests: An integrative review of interest research in vocational, organizational, and educational psychology. Journal of Vocational Behavior, 116 Part B, 103240. https://doi.org/10.1016/j.jvb.2018.10.016

  • The Psychometrics Centre. (n.d.a). Apply Magic Sauce. Retrieved July 8, 2017, from https://applymagicsauce.com/

  • The Psychometrics Centre. (n.d.b). Predicted traits. Retrieved July 8, 2017, from https://applymagicsauce.com/documentation_traits.html

  • Tonidandel, S., King, E. B., & Cortina, J. M. (2018). Big data methods: Leveraging modern data analytic techniques to build organizational science. Organizational Research Methods, 21, 525–547.

    Article  Google Scholar 

  • Tsacoumis, S., & Van Iddekinge, C. H. (2006). A comparison of incumbent and analyst ratings of O*NET skills. Human Resources Research Organization. Retrieved July 8, 2017, from https://www.onetcenter.org/dl_files/SkillsComp.pdf

  • U.S. Department of Labor, Employment and Training Administration. (2018). O*NET® Data Collection Program, Office of Management and Budget Clearance Package Supporting Statement. Part A: Justification. Author. Retrieved April 15, 2018, from https://www.onetcenter.org/dl_files/omb2018/Supporting_StatementA.pdf

  • U.S. Office of Personnel Management (2018). FedScope Employment Cube June 2018. Washington, D.C.: Author. Online: https://www.fedscope.opm.gov/employment.asp

  • Van Iddekinge, C. H., Roth, P. L., Putka, D. J., & Lanivich, S. E. (2011). Are you interested? A meta-analysis of relations between vocational interests and employee performance and turnover. Journal of Applied Psychology, 96, 1167–1194.

    Article  PubMed  Google Scholar 

  • Wegman, L. A., Hoffman, B. J., Carter, N. T., Twenge, J. M., & Guenole, N. (2018). Plotting job characteristics in context: Cross-temporal meta-analysis of changes in job characteristics since 1975. Journal of Management, 44, 352–386.

    Article  Google Scholar 

  • Wilson, M. A., Bennett, Jr., W., Gibson, S. G., & Alliger, G. M. (Eds.) (2012). The handbook of work analysis: Methods, systems, applications and science of work measurement in organizations. Routledge/Taylor & Francis Group.

  • Youyou, W., Kosinski, M., & Stillwell, D. (2015). Computer-based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences of the United States of America (PNAS), 112, 1036–1040.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Human Resources Research Organization, 66 Canal Center Plaza, Suite 700, Alexandria, VA, 22314-1578, USA

    Dan J. Putka & Martin C. Yu

  2. Department of Psychological Sciences, Rice University, 6100 Main Street – MS25, Houston, TX, 77005-1827, USA

    Frederick L. Oswald

  3. Department of Psychology, University of Minnesota, N218 Elliot Hall, 75 East River Road, Minneapolis, MN, 55455, USA

    Richard N. Landers

  4. Human Resources Research Organization, 1 Hacker Way, Menlo Park, CA, 94025, USA

    Adam S. Beatty

  5. Human Resources Research Organization, 700 North Hurstbourne Parkway, Suite 100, Louisville, KY, 40222-5393, USA

    Rodney A. McCloy

Authors
  1. Dan J. Putka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frederick L. Oswald
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard N. Landers
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adam S. Beatty
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rodney A. McCloy
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Martin C. Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dan J. Putka.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material. R code and key data files used in this article have been made publicly available via the Open Science Framework and can be accessed at https://osf.io/jbtkp/.

Supplementary file1 (XLSX 6505 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Putka, D.J., Oswald, F.L., Landers, R.N. et al. Evaluating a Natural Language Processing Approach to Estimating KSA and Interest Job Analysis Ratings. J Bus Psychol 38, 385–410 (2023). https://doi.org/10.1007/s10869-022-09824-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-022-09824-0

Keywords

Search

Navigation