Abstract
In changing accountability contexts, policymakers are engaging in international dialogue and collaborative efforts with new opportunities to reframe conversations about how to measure teacher quality and to (re) design and implement evaluation systems accordingly to ensure that they are fair, useable, feasible, and accurate. This study examined the lived experiences of school and district administrators in a large, fast-growth, suburban district in the USA regarding their districts’ new teacher evaluation system to better understand their perceptions of the system’s validity and reliability such that justifiable conclusions may be drawn about teachers’ effectiveness. Given concerns regarding validity and reliability, administrators generally discouraged external, high-stakes uses of evaluation results but valued the evaluation process and the data it provides for supporting teacher growth. As part of a larger study including teachers, findings can inform policymakers seeking to reform teacher evaluation frameworks to emphasize professional growth over high-stakes consequences.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adnot, M., Dee, T., Katz, V., & Wyckoff, J. (2017). Teacher turnover, teacher quality, and student achievement in DCPS. Educational Evaluation and Policy Analysis, 39(1), 54–76. https://doi.org/10.3102/0162373716663646.
Amrein-Beardsley, A. (2008). Methodological concerns about the education value-added assessment system. Educational Researcher, 37(2), 65–75. https://doi.org/10.3102/0013189X08316420.
Amrein-Beardsley, A. (2014). Rethinking value-added models in education: critical perspectives on tests and assessment-based accountability. New York, NY: Routledge.
Amrein-Beardsley, A., & Collins, C. (2012). The SAS education value-added assessment system (SAS® EVAAS®) in the Houston Independent School District (HISD): intended and unintended consequences. Education Policy Analysis Archives, 20. https://doi.org/10.14507/epaa.v20n12.2012.
Baker, E. L., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn, R. L., et al. (2010). Problems with the use of student test scores to evaluate teachers. Washington, DC: Economic Policy Institute Retrieved from http://www.epi.org/publication/bp278/. Accessed 5 May 2013.
Ballou, D., & Springer, M. G. (2015). Using student test scores to measure teacher performance: some problems in the design and implementation of evaluation systems. Educational Researcher, 44(2), 77–86. https://doi.org/10.3102/0013189X15574904.
Berliner, D. C. (2018). Between Scylla and Charybdis: reflections on and problems associated with the evaluation of teachers in an era of metrification. Education Policy Analysis Archives, 26(54). https://doi.org/10.14507/epaa.26.3820.
Borg, S. (2018). Teacher evaluation: global perspectives and their implications for English language teaching. A literature review. London, UK: British Council Retrieved from https://www.teachingenglish.org.uk/sites/teacheng/files/pub_Teacher_evaluation_Global_perspectives_implications_ELT.pdf. Accessed 21 Dec 2018.
Braun, H. I. (2005). Using student progress to evaluate teachers: a primer on value-added models. Princeton, NJ: Educational Testing Service Retrieved from http://www.ets.org/Media/Research/pdf/PICVAM.pdf. Accessed 5 May 2013.
Briggs, D., & Domingue, B. (2011). Due diligence and the evaluation of teachers: a review of the value-added analysis underlying the effectiveness rankings of Los Angeles Unified School District teachers by the Los Angeles Times. Boulder, CO: National Education Policy Center Retrieved from http://nepc.colorado.edu/publication/due-diligence. Accessed 4 Feb 2019.
Cannata, M., Rubin, M., Goldring, E., Grissom, J. A., Neumerski, C. M., Drake, T. A., & Schuermann, P. (2017). Using teacher effectiveness data for information-rich hiring. Educational Administration Quarterly, 53(2), 180–222. https://doi.org/10.1177/0013161X16681629.
Centre for Development and Enterprise (CDE). (2015a). Teacher evaluation: lessons from other countries [number 8]. Johannesburg, South Africa: Author. Retrieved from https://www.cde.org.za/teacher-evaluation-lessons-from-other-countries/. Accessed 22 Dec 2018.
Centre for Development and Enterprise (CDE). (2015b). Teacher evaluation in South African schools [number 9]. Johannesburg, South Africa: Author Retrieved from https://www.cde.org.za/teacher-evaluation-in-south-african-schools/. Accessed 22 Dec 2018.
Collins, C., & Amrein-Beardsley, A. (2014). Putting growth and value-added models on the map: a national overview. Teachers College Record, 116(1), 1-34. Retrieved from http://www.tcrecord.org/Content.asp?ContentId=17291. Accessed 04 Feb 2019.
Corcoran, S. P. (2010). Can teachers be evaluated by their students’ test scores? Should they be? The use of value-added measures of teacher effectiveness in policy and practice. Providence, RI: Annenberg Institute for School Reform Retrieved from http://www.annenberginstitute.org/publications/can-teachers-be-evaluated-their-students%E2%80%99-test-scores-should-they-be-use-value-added-me. Accessed 5 May 2013.
Cuevas, R., Ntoumanis, N., Fernandez-Bustos, J. G., & Bartholomew, K. (2018). Does teacher evaluation based on student performance predict motivation, well-being, and ill-being? Journal of School Psychology, 68, 154–162. https://doi.org/10.1016/j.jsp.2018.03.005.
Danielson, C. (2013). The framework for teaching evaluation instrument. Princeton, NJ: The Danielson Group Retrieved from http://danielsongroup.org/download/?download=448. Accessed 4 Feb 2019.
Darling-Hammond, L. (2013). Getting teacher evaluation right: what really matters for effectiveness and improvement. New York, NY: Teachers College.
Darling-Hammond, L. (2015). Can value added add value to teacher evaluation? Educational Researcher, 44(2), 132–137. https://doi.org/10.3102/0013189X15575346.
Darling-Hammond, L., Amrein-Beardsley, A., Haertel, E., & Rothstein, J. (2012). Evaluating teacher evaluation. Phi Delta Kappan, 93(6), 8–15. https://doi.org/10.1177/003172171209300603.
Derrington, M. L. (2016). Implementing teacher evaluation: lattice of leadership. Journal of Research on Leadership Education, 11(2), 181–199. https://doi.org/10.1177/1942775116658689.
Derrington, M. L., & Campbell, J. W. (2017). Teacher evaluation policy tools: principals’ selective use in instructional leadership. Leadership and Policy in Schools., 17, 568–590. https://doi.org/10.1080/15700763.2017.1326143.
Derrington, M. L., & Campbell, J. W. (2018). High-stakes teacher evaluation policy: US principals’ perspectives and variations in practice. Teachers and Teaching: Theory and Practice, 24(3), 246–263. https://doi.org/10.1080/13540602.2017.1421164.
Dodson, R. L. (2017). An analysis of principals’ perceptions of the primary teaching evaluation system used in eight U.S. states. International Journal of Education Policy and Leadership, 12(5), 1–22. https://doi.org/10.22230/ijelp.2017v12n5a773.
Donaldson, M. L. (2011). Principals’ approaches to developing teacher quality: constraints and opportunities in hiring, assigning, evaluating, and developing teachers. Washington, DC: Center for American Progress Retrieved from http://cdn.americanprogress.org/wp-content/uploads/issues/2011/02/pdf/principal_report.pdf. Accessed 8 Aug 2013.
Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (pp. 119–161). New York, NY: Macmillan.
Every Student Succeeds Act (ESSA) of 2015, Pub. L. No. 114-95, § 129 Stat. 1802. (2015).
Fink, A. (1995). Evaluation for education and psychology. Thousand Oaks, CA: Sage.
Finster, M., & Milanowski, A. (2018). Teacher perceptions of a new performance evaluation system and their influence on practice: a within- and between-school level analysis. Education Policy Analysis Archives, 26(41). https://doi.org/10.14507/epaa.26.3500.
Flores, M. A., & Derrington, M. L. (2017). School principals’ views of teacher evaluation policy: lessons learned from two empirical studies. International Journal of Leadership in Education, 20(4), 416–431. https://doi.org/10.1080/13603124.2015.1094144.
Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: strategies for qualitative research. Chicago, IL: Aldine.
Goldring, E., Grissom, J. A., Rubin, M., Neumerski, C. M., Cannata, M., Drake, T., & Schuermann, P. (2015). Make room value added: principals’ human capital decisions and the emergence of teacher observation data. Educational Researcher, 44(2), 96–104. https://doi.org/10.3102/0013189X15575031.
Harris, D. N. (2011). Value-added measures in education: what every educator needs to know. Cambridge, MA: Harvard Education Press.
Harris, D. N., Ingle, W. K., & Rutledge, S. A. (2014). How teacher evaluation methods matter for accountability: a comparative analysis of teacher effectiveness ratings by principals and teacher value-added measures. American Educational Journal, 51(1), 73–112. https://doi.org/10.3102/0002831213517130.
Hazi, H. M. (2017). VAM under scrutiny: teacher evaluation litigation in the states. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 90(5–6), 184–190. https://doi.org/10.1080/00098655.2017.1366803.
Herlihy, C., Karger, E., Pollard, C., Hill, H. C., Kraft, M. A., Williams, M., & Howard, S. (2014). State and local efforts to investigate the validity and reliability of scores from teacher evaluation systems. Teachers College Record, 116, 1–28.
Hopkins, P. (2016). Teacher voice: how teachers perceive evaluations and how leaders can use this knowledge to help teachers grow professionally. NASSP Bulletin, 100(1), 5–25. https://doi.org/10.1177/0192636516670771.
Hopkins, E., Hendry, H., Garrod, F., McClare, S., Pettit, D., Smith, L., Burrell, H., & Temple, J. (2016). Teachers’ views of the impact of school evaluation and external inspection processes. Improving Schools, 19(1), 52–61. https://doi.org/10.1177/1365480215627894.
Jiang, J. Y., Sporte, S. E., & Luppescu, S. (2015). Teacher perspectives on evaluation reform: Chicago’s REACH students. Educational Researcher, 44(2), 105–116. https://doi.org/10.3102/0013189X15575517.
Johnson, S. M. (2005). The prospects for teaching as a profession. In L. V. Hedges & B. Schneider (Eds.), The social organization of schooling (pp. 72–90). New York, NY: Russell Sage Foundation.
Joint Committee on Standards for Educational Evaluation. (2008). The personnel evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.
Kane, M. T. (2008). Terminology, emphasis, and utility in validation. Educational Researcher, 37(2), 76–82. https://doi.org/10.3102/0013189X08315390.
Kraft, M. A., & Gilmour, A. F. (2016). Can principals promote teacher development as evaluators? A case study of principals’ views and experiences. Educational Administration Quarterly, 52(5), 711–753. https://doi.org/10.1177/001316IXI16653445.
Kraft, M. A., & Gilmour, A. F. (2017). Revisiting the widget effect: teacher evaluation reforms and the distribution of teacher effectiveness. Educational Researcher, 46(5), 234–249. https://doi.org/10.3102/0013189X17718797.
Lavigne, A. L. (2014). Exploring the intended and unintended consequences of high-stakes teacher evaluation on schools, teachers, and students. Teachers College Record, 116, 1–29 Retrieved from http://www.tcrecord.org/library/abstract.asp?contentid=17294. Accessed 24 Jan 2015.
Lavigne, A. L., & Chamberlain, R. W. (2017). Teacher evaluation in Illinois: school leaders’ perceptions and practices. Educational Assessment, Evaluation and Accountability, 29, 179–209. https://doi.org/10.1007/s11092-016-9250-0.
Lavigne, A. L., & Good, T. L. (2014). Teacher and student evaluation: moving beyond the failure of school reform. New York, NY: Routledge.
Liu, S., & Zhao, D. (2013). Teacher evaluation in China: latest trends and future directions. Educational Assessment, Evaluation and Accountability, 25(3), 231–250. https://doi.org/10.1007/s11092-013-9168-8.
Loewus, L. (2017). Are states changing course on teacher evaluation? Test-score growth plays lesser role in six states. Education Week, 37(13), 1–17 Retrieved from https://www.edweek.org/ew/articles/2017/11/15/are-states-changing-course-on-teacher. Accessed 28 Feb 2018.
Martinez, F., Taut, S., & Schaaf, K. (2016). Classroom observation for evaluating and improving teaching: an international perspective. Studies in Educational Evaluation, 49, 15–29. https://doi.org/10.1016/j.stueduc.2016.03.002.
Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955–966. https://doi.org/10.1037/0003-066X.30.10.955.
Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35(11), 1012–1027. https://doi.org/10.1037/0003-066X.35.11.1012.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage Publications.
No Child Left Behind (NCLB) Act of 2001, Public Law 107–110, § 115 Stat. 1425. (2002).
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw Hill.
O’Pry, S. C., & Schumacher, G. (2012). New teachers’ perceptions of a standards-based performance appraisal system. Educational Assessment, Evaluation and Accountability, 24(4), 325–350. https://doi.org/10.1007/s11092-012-9148-4.
Organisation for Economic Co-operation and Development (OECD). (2013). Teachers for the 21st century: Using evaluation to improve teaching. Paris: OECD Publishing.
Papay, J. P. (2010). Different tests, different answers: the stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48(1), 163–193. https://doi.org/10.3102/00002831210362589.
Paufler, N. A. (2018a). Declining morale, diminishing autonomy, and decreasing value: principal reflections on a high-stakes teacher evaluation system. International Journal of Education Policy & Leadership, 13(8). https://doi.org/10.22230/ijepl.2018v13n8a831.
Paufler, N. A. (2018b). The value a teacher evaluation system adds in practice: school administrator and teacher perceptions of system effectiveness. Manuscript submitted for publication.
Popham, W. J. (1988). Educational evaluation (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
Reddy, L. A., Dudek, C. M., Peters, S., Alperin, A., Kettler, R. J., & Kurz, A. (2018). Teachers’ and school administrators’ attitudes and beliefs of teacher evaluation: a preliminary investigation of high poverty school districts. Educational Assessment, Evaluation and Accountability, 30, 47–40. https://doi.org/10.1007/s11092-017-9263-3.
Reid, D. B. (2018). School principals acting as middle leaders implementing new teacher evaluation systems. School Leadership & Management., 1–17. https://doi.org/10.1080/13632434.2018.1508013.
Saldaña, J. (2013). The coding manual for qualitative researchers (2nd ed.). London: Sage.
Smith, M. L. (1997). Mixing and matching: methods and models. New Directions for Evaluation, 74, 73–85. https://doi.org/10.1002/ev.1073.
Stake, R. E. (1978). The case study method in social inquiry. Educational Researcher, 7(2), 5–8. https://doi.org/10.3102/0013189X007002005.
Stake, R. E., & Trumbull, D. (1982). Naturalistic generalizations. Review Journal of Philosophy and Social Science, 7, 1–12.
Stewart, V. (2013). Teacher quality: the 2013 International Summit on the Teaching Profession. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/files/teachingsummit2013.pdf. Accessed 21 Dec 2018.
Stewart, V. (2015). Implementing highly effective teacher policy and practice. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/files/2015-istp-report.pdf. Accessed 22 Dec 2018.
Stewart, V. (2016). Teachers’ professional learning and growth: creating the conditions to achieve quality teaching for excellent learning outcomes. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/files/2016-istp-report-small.pdf. Accessed 22 Dec 2018.
Stewart, V. (2018). New challenges and opportunities facing the teaching profession in public education. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/sites/default/files/inline-files/2018-international-summit-on-the-teaching-profession-edu-istp.pdf.
Strauss, A. L., & Corbin, J. (1995). Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage Publications.
Taut, S., & Sun, Y. (2014). The development and implementation of a national, standards-based, multi-method teacher performance assessment system in Chile. Education Policy Analysis Archives, 22(71). https://doi.org/10.14507/epaa.v22n71.2014.
Taylor, E. S., & Tyler, J. H. (2012). Can teacher evaluation improve teaching? Education Next, 12(4) Retrieved from https://www.educationnext.org/can-teacher-evaluation-improve-teaching/. Accessed 4 Feb 2019.
Tucker, P. D., & Stronge, J. H. (2005). Linking teacher evaluation and student learning. Alexandria, VA: Association for Supervision and Curriculum Development.
United Nations Educational, Scientific and Cultural Organization (UNESCO). (2016). Education 2030. Incheon declaration and framework for action. Retrieved from https://unesdoc.unesco.org/ark:/48223/pf0000245656. Accessed 22 Dec 2018.
United States Department of Education. (2009). Race to the Top program: executive summary. Retrieved from http://www2.ed.gov/programs/racetothetop/executive-summary.pdf. Accessed 6 July 2013.
United States Department of Education. (2010). Teacher Incentive Fund. Retrieved from http://www2.ed.gov/programs/teacherincentive/index.html. Accessed 6 July 2013.
Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: our national failure to acknowledge and act of differences in teacher effectiveness (2nd ed.). Brooklyn, NY: The New Teacher Project (TNTP). Retrieved from http://tntp.org/ideas-and-innovations/view/the-widget-effect. Accessed 31 Jan 2018.
Will, M. (2016). Assessing quality of teaching staff still complex despite ESSA’s leeway. Education Week, 36(16), 31–32 Retrieved from http://www.edweek.org/ew/articles/2017/01/04/assessing-quality-of-teaching-staff-still-complex.html?intc=EW-QC17-TOC&_ga=1.138540723.1051944855.1481128421. Accessed 9 Apr 2017.
World Bank. (2013). What matters most in teacher policies? A framework for building a more effective teaching profession. Retrieved from https://openknowledge.worldbank.org/bitstream/handle/10986/20143/901820NWP0no4000Box385307B00PUBLIC0.pdf?sequence=1&isAllowed=y. Accessed 4 Feb 2019.
World Bank. (2018). World development report: learning to realize education’s promise. Washington, DC: World Bank Retrieved from www.worldbank.org/en/publication/wdr2018. Accessed 4 Feb 2019.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Paufler, N.A., Clark, C. Reframing conversations about teacher quality: school and district administrators’ perceptions of the validity, reliability, and justifiability of a new teacher evaluation system. Educ Asse Eval Acc 31, 33–60 (2019). https://doi.org/10.1007/s11092-019-09292-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11092-019-09292-w