How Students’ Intellectual Orientations and Cognitive Reasoning Abilities and May Shape Their Perceptions of Good Teaching Practices

Abstract

Recent research has uncovered significant concerns about the validity of some types of college student self-reports. This study examines the extent to which student reports about a critical type of college experience—good teaching practices—may be biased as a function of students’ intellectual orientations and cognitive reasoning abilities. Perceptions of instruction and instructional practices are especially important in higher education, given their increasing use for institutional quality assurance, as well as faculty rehiring and promotion processes. Using a large, multi-institutional, longitudinal dataset of first-year students, this study shows that several cognitive indicators predict perceptions of six different sets of good teaching practices and that these relationships do not seem to be explained by actual differences in students’ experiences. Additional analyses indicate that halo effects, in which global evaluations of instructor quality and institutional satisfaction affect students’ perceptions of their engagement with good practices, may partially explain these findings. The results provide important implications for practice and research related to college student survey data, including ways that these biases can be reduced or eliminated to more accurately capture students’ engagement in good practices and the factors that may contribute to students’ perceptions of their environment.

Appendices

Appendix A Variable descriptions, descriptive statistics, and reliabilities for ACT sample (N = 6531).

Variable	Definition	Mean	Standard deviation	Min	Max
Intellectual orientations and cognitive reasoning abilities
Precollege academic motivation	Mean-based scale measuring academic motivation; 8-item scale, α = 69; standardized	0.00	1.00	−4.47	2.50
Precollege need for cognition	The degree to which one enjoys engaging in effortful cognitive activities; 18-item scale, α = 0.89; standardized	0.00	1.00	−3.82	2.50
Precollege ability (ACT or equivalent)	Composite ACT or SAT equivalent score converted to an ACT metric standardized	0.00	1.00	−3.77	2.22
Precollege critical thinking	Critical thinking skills; 32-item scale, α = 0.81—0.82; standardized	0.00	1.00	−2.85	1.94
Good teaching practices
Student-faculty interactions	Students’ perceptions of faculty's interest and willingness to interact with students outside the classroom and the frequency with which students interacted with faculty outside the classroom; 9-item scale, α = 0.77; standardized	0.00	1.00	−3.67	2.85
Active learning	Students’ perceptions of the frequency that faculty used various active learning techniques (including class discussions, class presentations, and assignments that required critique of an argument); 10-item scale, α = 0.77; standardized	0.00	1.00	−3.63	2.34
Collaborative learning	Students' perceptions of the frequency that faculty integrated collaborative learning approaches inside and outside of class (i.e., group projects, study groups); 7-item scale, α = 0.71; standardized	0.00	1.00	−3.50	2.63
Prompt feedback	Students’ perceptions that faculty provided timely written or oral feedback and evaluated student learning informally in the classroom; 3-item scale, α = 0.67; standardized	0.00	1.00	−3.00	2.27
Time on task	Students’ perceptions that instruction was relevant, organized, and helpful for achieving clearly defined course goals and the amount of effort students put forth on studying as a result of institutional and instructor expectations; 8-item scale, α = 0.72; standardized	0.00	1.00	−5.91	2.49
High expectations	Students’ perceptions of the frequency that faculty challenged students intellectually, especially through the use of techniques requiring higher-order thinking (i.e., applying, critiquing, and/or arguing); 5-item scale, α = 0.81; standardized	0.00	1.00	−3.36	2.00
Potential mechanisms
Perceptions of instructor quality	Generalized assessment of teaching ability and interest in teaching among instructors; 2-item scale, α = 0.89; standardized	0.00	1.00	−3.94	1.41
Overall college satisfaction	Overall satisfaction with educational experience at this institution; 2-item scale, α = 0.72; standardized	0.00	1.00	−3.68	1.10
Student background characteristics
Race/Ethnicity: Black; African American	0 = no; 1 = yes	0.09	0.28	0.00	1.00
Race/Ethnicity: Asian; Pacific Islander	0 = no; 1 = yes	0.05	0.23	0.00	1.00
Race/Ethnicity: Latinx	0 = no; 1 = yes	0.05	0.21	0.00	1.00
Race/Ethnicity: Other; Race/Ethnicity Unknown	0 = no; 1 = yes	0.03	0.18	0.00	1.00
Sex: Male	0 = no; 1 = yes	0.38	0.48	0.00	1.00
First-Generation Student (no parent attended college)	0 = no; 1 = yes	0.11	0.31	0.00	1.00
H.S. Interactions with Teachers	Frequency of interacting with teachers outside of class during high school; (1 = never, to 5 = very often); standardized	0.00	1.00	−2.06	0.49
First-year college experiences
Paradigmatic development of courses taken		0.00	1.00	−1.48	2.68
Intended major: biological sciences	0 = no; 1 = yes	0.11	0.31	0.00	1.00
Intended major: business	0 = no; 1 = yes	0.11	0.32	0.00	1.00
Intended major: education	0 = no; 1 = yes	0.07	0.26	0.00	1.00
Intended major: engineering	0 = no; 1 = yes	0.05	0.21	0.00	1.00
Intended major: physical science	0 = no; 1 = yes	0.06	0.24	0.00	1.00
Intended major: professional	0 = no; 1 = yes	0.11	0.32	0.00	1.00
Intended major: social science	0 = no; 1 = yes	0.17	0.37	0.00	1.00
Courses: diverse cultures and perspectives (e.g., ethnic studies)	Number of courses taken focusing on diverse cultures and perspectives (1 = 0 courses, to 5 = 4 or more courses)	0.00	1.00	−0.71	3.67
Courses: women’s/gender studies	Number of courses taken focusing on women’s/gender studies (1 = 0 courses, to 5 = 4 or more courses)	0.00	1.00	−0.42	5.86
Courses: focus on issues of equality and/or social justice	Number of courses taken focusing on issues of equality and/or social justice (1 = 0 courses, to 5 = 4 or more courses)	0.00	1.00	−0.67	4.14
Honors program or college	0 = no; 1 = yes	0.16	0.37	0.00	1.00
First-year seminar	0 = no; 1 = yes	0.67	0.47	0.00	1.00
Learning community	0 = no; 1 = yes	0.32	0.46	0.00	1.00
Service learning	0 = no; 1 = yes	0.45	0.59	0.00	1.00
Undergraduate research	0 = no; 1 = yes	0.05	0.22	0.00	1.00
College grades	Students’ self-reported grades (1 = C- or lower, to 8 = A)	0.00	1.00	−3.24	1.22
Hours working for pay	How many hours per week students spend working for pay (1 = 0 h to 8 = more than 30 h)	0.00	1.00	−0.68	8.12

Appendix B Items included in each scale of good teaching practices

Faculty-student interactions (a = 0.76)
Most faculty with whom I have had contact are genuinely interested in studentsa
Most faculty with whom I have had contact are willing to spend time outside of class to discuss issues of interest and importance to studentsa
Most faculty with whom I have had contact are interested in helping students grow in more than just academic areasa
During current school year, how often have you discussed grades or assignments with an instructor?b
During current school year, how often have you talked about career plans with a faculty member or advisor?b
During current school year, how often have you discussed ideas from readings or classes with faculty members outside of class?b
During current school year, how often have you worked with faculty members on activities other than coursework (committees, orientation, student life activities, etc.)?b
Indicate the extent to which you agree/disagree that I am satisfied with the opportunities to meet and interact informally with faculty membersa

Collaborative learning (a = 0.70)
In my classes, students taught each other in addition to faculty teachingc
Faculty encouraged me to participate in study groups outside of classc
I have participated in one or more study group(s) outside of class. c
In your experience at your institution during the current school year, about how often have you worked with classmates outside of class to prepare class assignments?b

Active learning (a = 0.73)
In your experience at your institution during the current school year, about how often have you asked questions in class or contributed to class discussions?b
In your experience at your institution during the current school year, about how often have you made a class presentation?b
During the current school year, how much has your coursework emphasized synthesizing and organizing ideas, information, or experiences into new, more complex interpretations and relationships?d
During the current school year, how much has your coursework emphasized making judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions?d
How often have exams or assignments required me to write essaysc
How often have exams or assignments required me to use course content to address a problem not presented in the coursec
How often have exams or assignments required me to compare or contrast topics or ideas from a coursec
How often have exams or assignments required me to point out the strengths and weaknesses of a particular argument or point of viewc
How often have exams or assignments required me to argue for or against a particular point of view and defend my argument?c

Time on task (a = 0.72)
About how many hours in a typical week do you spend preparing for class (studying, reading, writing, doing homework or lab work, analyzing data, rehearsing, and other academic activities)?f
Frequency that faculty gave assignments that helped in learning the course materialc
Frequency that class time was used effectivelyc
Frequency that the presentation of material was well organizedc
Frequency that course goals and requirements were clearlyc explained

High expectations (a = 0.71)
How often have faculty asked challenging questions in class?c
How often have faculty asked you to argue for or against a particular point of view?c
How often have faculty challenged your ideas in class?c
Mark the box that best represents the extent to which your examinations during the current school year challenged you to do your best worke
In your experience at your institution during the current school year, about how often have you worked harder than you thought you could to meet an instructor's standards or expectations?b
How often have students challenged each other's ideas in class?c

Prompt feedback (a = 0.67)
How often have faculty informed you of your level of performance in a timely manner?c
How often have faculty checked to see if you had learned the material well before going on to new material?c
In your experience at your institution during the current school year, about how often have you received prompt written or oral feedback from faculty on your academic performance?b

1. Response scales are as follows:
2. ^a1 = strongly disagree to 5 = strongly agree
3. ^b1 = never to 4 = very often
4. ^c1 = never to 5 = very often
5. ^d1 = very little to 4 = very much
6. ^e1 = very little to 7 = very much
7. ^f1 = 0 h to 8 = more than 30 hours