Abstract
Researchers, assessment professionals, and faculty in higher education increasingly depend on survey data from students to make pivotal curricular and programmatic decisions. The surveys collecting these data often require students to judge frequency (e.g., how often), quantity (e.g., how much), or intensity (e.g., how strongly). The response options given for these questions are usually vague and include responses such as “never,” “sometimes,” and “often.” However, the meaning that respondents give to these vague responses may vary. This study aims to determine the efficacy of using vague quantifiers in survey research. More specifically, the purpose of this study is to explore the meaning that respondents ascribe to vague response options and whether or not those meanings vary by student characteristics. Results from this study indicate a high degree of correspondence between vague and numeric response and suggest that students seem to adapt the meaning of “sometimes,” “often,” and “very often” based on the appropriate reference for the question. Overall, findings provide evidence of the utility and appropriateness of using vague response options. Some differences by student characteristics and the implications of these differences are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Al Baghal, T. (2014). Is vague valid? The comparative predictive validity of vague quantifiers and numeric response options. Survey Research Methods,8(3), 169–179.
Algina, J., Keselman, H. J., & Penfield, R. D. (2005). An alternative to Cohen’s standardized mean difference effect size: A robust parameter and confidence interval in the two independent groups case. Psychological Methods,10, 317–328.
Anaya, G. (1999). College impact on student learning: Comparing the use of self-reported gains, standardized test scores, and college grades. Research in Higher Education,40, 499–526.
Bradburn, N. M., & Miles, C. (1979). Vague quantifiers. Public Opinion Quarterly,43(1), 92–101.
Brown, N. R. (1997). Context memory and the selection of frequency estimation strategies. Journal of Experimental Psychology: Learning, Memory, and Cognition,23(4), 898–914.
Cole, J. S., & Korkmaz, A. (2013). Estimating college student behavior frequencies: Do vague and enumerated estimation strategies yield similar results? Journal of Applied Research in Higher Education,5(1), 58–71.
Drummond, R. J., Sheperis, C. J., & Jones, K. D. (2016). Assessment procedures for counselors and helping professionals (8th ed.). Boston, MA: Pearson.
Dumford, A. D., & Miller, A. L. (2018). Online learning in higher education: Exploring advantages and disadvantages for engagement. Journal of Computing in Higher Education,30(3), 452–465. https://doi.org/10.1007/s12528-018-9179-z.
Fosnacht, K. F., Sarraf, S. A., Howe, E., & Peck, L. K. (2017). How important are high response rates for college surveys?”. The Review of Higher Education,40, 245–265.
Hayek, J. C., Carini, R. M., O’Day, P. T., & Kuh, G. D. (2002). Triumph or tragedy: Comparing student engagement levels of members of Greek-letter organizations and other students. Journal of College Student Development,43, 643–663.
Institute of International Education. (2016). International student enrollments by institutionaltype, 2014/15–2015/16. Open Doors Report on International Educational Exchange. Retrieved September 28, 2018, from https://www.iie.org/opendoors.
Krumpal, I., Rauhut, H., Böhr, D., & Naumann, E. (2016). How likely is ‘likely’? Subjective perception and communication of victimization probabilities. Methods, Data, Analyses,2(1), 3–27.
Lambert, A. D., & Miller, A. L. (2014). Lower response rates on alumni surveys might not mean lower response representativeness. Educational Research Quarterly,37(3), 38–51.
Lenzner, T., Kaczmirek, L., & Lenzner, A. (2010). Cognitive burden of survey questions and response times: a psycholinguistic experiment. Applied Cognitive Psychology,24(7), 1003–1020.
Mair, P., & Wilcox, R. (2017). WRS2: A collection of robust statistical methods. R package version 0.9-2.
Mayhew, M. J., Rockenbach, A. N., Bowman, N. A., Seifert, T. A., Wolniak, G. C., Pascarella, E. T., et al. (2016). How college affects students: 21st century evidence that higher education works. San Francisco: Jossey-Bass.
McCormick, A. C., Kinzie, J., & Gonyea, R. M. (2013). Student engagement: Bridging research and practice to improve the quality of undergraduate education. Higher Education: Handbook of Theory and Research, 47–92.
National Center for Education Statistics. (2018). Digest of education statistics, 2016. Table 303.10. Washington, D.C.: U.S. Department of Education.
National Survey of Student Engagement. (2016). NSSE 2016 overview. Bloomington, IN: Center for Postsecondary Research. Retrieved March 23, 2017, from https://nsse.indiana.edu/2016_Institutional_Report/pdf/NSSE_Overview_2016.pdf.
National Survey of Student Engagement. (2019). An explanation of Weighting in the NSSE Institutional Report. Retrieved August 29, 2019, from https://nsse.indiana.edu/html/weighting.cfm.
Nelson Laird, T. F., Korkmaz, A., & Chen, P. (2008). How often is “often” revisited: The meaning and linearity of vague quantifiers used on the national survey of student engagement. Presented at Annual Meeting of AERA. Retrieved January 10, 2018.
Pace, R., & Friedlander, J. (1982). The meaning of response categories: How often is "occasionally," "often," and "very often"? Research in Higher Education,17(3), 267–281.
Picciano, A. G. (2012). The evolution of big data and learning analytics in American higher education. Online Learning,16(3), 9–20.
Pike, G. R. (1991). The effects of background, coursework, and involvement on students’ grades and satisfaction. Research in Higher Education,32, 16–30.
Porter, S. R. (2011). Do college student surveys have any validity? The Review of Higher Education,35(1), 45–76. https://doi.org/10.1353/rhe.2011.0034.
Reyna, V. F., & Brainerd, C. J. (2008). Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learning and Individual Differences,18(1), 89–107.
Salkind, N. J. (2010). Encyclopedia of research design. Thousand Oaks, CA: SAGE.
Schaeffer, N. C. (1991). Hardly ever or constantly? Group comparisons using vague quantifiers. Public Opinion Quarterly,55(3), 395–423.
Schneider, S., & Stone, A. A. (2016). The meaning of vaguely quantified frequency response options on a quality of life scale depends on respondents’ medical status and age. Quality of Life Research,25(10), 2511–2521.
Seaman, J. E., Allen, I. E., & Seaman, J. (2018). Grade increase: Tracking distance education in the United States. BABSON Survey Research Group. Retrieved April 3, 2018, from https://www.onlinelearningsurvey.com/highered.html.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.
Wänke, M. (2002). Conversational norms and the interpretation of vague quantifiers. Applied Cognitive Psychology,16(3), 301–307. https://doi.org/10.1002/acp.787.
Wilcox, R. R. (2012). Introduction to robust estimation and hypothesis testing (3rd ed.). Amsterdam: Elsevier.
Yuen, K. K. (1974). The two sample trimmed t for unequal population variances. Biometrika,61, 165–170.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Rocconi, L.M., Dumford, A.D. & Butler, B. Examining the Meaning of Vague Quantifiers in Higher Education: How Often is “Often”?. Res High Educ 61, 229–247 (2020). https://doi.org/10.1007/s11162-020-09587-8
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11162-020-09587-8