Abstract
Few studies have engaged in data-driven investigations of the presence, or frequency, of what could be considered retaliatory assessor behaviour in Multi-source Feedback (MSF) systems. In this study, authors explored how assessors scored others if, before assessing others, they received their own assessment score. The authors examined assessments from an established MSF system in which all clinical team members - medical students, interns, residents, fellows, and supervisors - anonymously assessed each other. The authors identified assessments in which an assessor (i.e., any team member providing a score to another) gave an aberrant score to another individual. An aberrant score was defined as one that was more than two standard deviations from the assessment receiver’s average score. Assessors who gave aberrant scores were categorized according to whether their behaviour was preceded by: (1) receiving a score or not from another individual in the MSF system (2) whether the score they received was aberrant or not. The authors used a multivariable logistic regression model to investigate the association between the type of score received and the type of score given by that same individual. In total, 367 unique assessors provided 6091 scores on the performance of 484 unique individuals. Aberrant scores were identified in 250 forms (4.1%). The chances of giving an aberrant score were 2.3 times higher for those who had received a score, compared to those who had not (odds ratio 2.30, 95% CI:1.54–3.44, P < 0.001). Individuals who had received an aberrant score were also 2.17 times more likely to give an aberrant score to others compared to those who had received a non-aberrant score (2.17, 95% CI:1.39–3.39, P < 0.005) after adjusting for all other variables. This study documents an association between receiving scores within an anonymous multi-source feedback (MSF) system and providing aberrant scores to team members. These findings suggest care must be given to designing MSF systems to protect against potential downstream consequences of providing and receiving anonymous feedback.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Association, A. P., Association, A. E. R., & Education, N. C. (1966). Standards for educational and psychological tests and manuals. American Psychological Association. o. M. i.
Baker, K., Haydar, B., & Mankad, S. (2017). A feedback and evaluation system that provokes minimal retaliation by trainees. Anesthesiology, 126(2), 327–337. https://doi.org/10.1097/aln.0000000000001471.
Brett, J. F., & Atwater, L. E. (2001). 360 degree feedback: Accuracy, reactions, and perceptions of usefulness. Journal Of Applied Psychology, 86(5), 930–942. https://doi.org/10.1037/0021-9010.86.5.930.
Bullock, A. D., Hassell, A., Markham, W. A., Wall, D. W., & Whitehouse, A. B. (2009). How ratings vary by staff group in multi-source feedback assessment of junior doctors. Medical Education, 43(6), 516–520. https://doi.org/10.1111/j.1365-2923.2009.03333.x.
Caretta-Weyer, H. A., Kraut, A. S., Kornegay, J. G., & Yarris, L. M. (2017). The View from over Here: A Framework for Multi-Source Feedback. J Grad Med Educ, 9(3), 367–368. https://doi.org/10.4300/jgme-d-17-00200.1.
Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3), 1–58.
Cook, D. A., Zendejas, B., Hamstra, S. J., Hatala, R., & Brydges, R. (2014). What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Advances in Health Sciences Education, 19, 233–250.
Dory, V., Danoff, D., Plotnick, L. H., Cummings, B. A., Gomez-Garibello, C., Pal, N. E., & Young, M. (2021). Does Educational Handover Influence subsequent Assessment? Academic Medicine, 96(1), 118–125. https://doi.org/10.1097/acm.0000000000003528.
Engdahl, R. A., Keating, R. J., & Perrachione, J. (1993). Effects of grade feedback on student evaluation of instruction. Journal of Management Education, 17(2), 174–184.
Eva, K. W., Armson, H., Holmboe, E., Lockyer, J., Loney, E., Mann, K., & Sargeant, J. (2012). Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Advances in Health Sciences Education, 17(1), 15–26. https://doi.org/10.1007/s10459-011-9290-7.
Gardner, A. K., & Scott, D. J. (2016). Repaying in Kind: Examination of the reciprocity effect in Faculty and Resident evaluations. Journal Of Surgical Education, 73(6), e91–e94. https://doi.org/10.1016/j.jsurg.2016.04.015.
Gauthier, G., St-Onge, C., & Tavares, W. (2016). Rater cognition: Review and integration of research findings. Medical Education, 50(5), 511–522. https://doi.org/10.1111/medu.12973.
Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the ‘black box’ differently: assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068. https://doi.org/10.1111/medu.12546.
Gorth, D. J., Magee, R. G., Rosenberg, S. E., & Mingioni, N. (2021). Gender disparity in evaluation of Internal Medicine Clerkship performance. JAMA Network Open, 4(7), e2115661–e2115661. https://doi.org/10.1001/jamanetworkopen.2021.15661.
Govaerts, M. J., Schuwirth, L. W., Van der Vleuten, C. P., & Muijtjens, A. M. (2011). Workplace-based assessment: Effects of rater expertise. Adv Health Sci Educ Theory Pract, 16(2), 151–165. https://doi.org/10.1007/s10459-010-9250-7.
Govaerts, M. J., Van de Wiel, M. W., Schuwirth, L. W., Van der Vleuten, C. P., & Muijtjens, A. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Adv Health Sci Educ Theory Pract, 18(3), 375–396. https://doi.org/10.1007/s10459-012-9376-x.
Hayward, M. F., Curran, V., Curtis, B., Schulz, H., & Murphy, S. (2014). Reliability of the Interprofessional Collaborator Assessment Rubric (ICAR) in Multi Source Feedback (MSF) with post-graduate medical residents. BMC Medical Education, 14(1), 1049. https://doi.org/10.1186/s12909-014-0279-9.
Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682. https://doi.org/10.3109/0142159x.2010.500704.
Humphrey-Murto, S., Shaw, T., Touchie, C., Pugh, D., Cowley, L., & Wood, T. J. (2021). Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment. Adv Health Sci Educ Theory Pract, 26(3), 1133–1156. https://doi.org/10.1007/s10459-021-10032-3.
Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. (2011). Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education, 45(10), 1048–1060. https://doi.org/10.1111/j.1365-2923.2011.04025.x.
Labbé, M., Young, M., & Nguyen, L. H. (2018). Validity evidence as a key marker of quality of technical skill assessment in OTL–HNS. The Laryngoscope, 128(10), 2296–2300.
Mann, K., van der Vleuten, C., Eva, K., Armson, H., Chesluk, B., Dornan, T., & Sargeant, J. (2011). Tensions in informed self-assessment: How the desire for feedback and reticence to collect and use it can conflict. Academic Medicine, 86(9), 1120–1127. https://doi.org/10.1097/ACM.0b013e318226abdd.
Marceau, M., Gallagher, F., Young, M., & St-Onge, C. (2018). Validity as a social imperative for assessment in health professions education: A concept analysis. Medical Education, 52(6), 641–653.
Messick, S. (1989). Validity. em r. linn (org.), educational measurement.(13–103). New York, NY: American Council on Education and Macmillan Publishing Company.
Mirzazadeh, A., Mortaz Hejri, S., Jalili, M., Asghari, F., Labaf, A., Sedaghat Siyahkal, M., & Saleh, N. (2014). Defining a competency framework: The first step toward competency-based medical education. Acta Med Iran, 52(9), 710–716.
Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29(9), 855–871. https://doi.org/10.1080/01421590701775453.
Paget, M., Brar, G., Veale, P., Busche, K., Coderre, S., Woloschuk, W., & McLaughlin, K. (2018). The grades that clinical teachers give students modifies the grades they receive. Adv Health Sci Educ Theory Pract, 23(2), 241–247. https://doi.org/10.1007/s10459-017-9783-0.
Richardson, R. D., & Williams, R. L. (2021). Linkages between Grade Point Average and Student ratings. Journal of Educational Research and Practice, 11(1), 16–33.
Richmond, M., Canavan, C., Holtman, M. C., & Katsufrakis, P. J. (2011). Feasibility of implementing a standardized multisource feedback program in the graduate medical education environment. J Grad Med Educ, 3(4), 511–516. https://doi.org/10.4300/jgme-d-10-00088.1.
Robins, L., Smith, S., Kost, A., Combs, H., Kritek, P. A., & Klein, E. J. (2020). Faculty perceptions of formative feedback from medical students. Teaching And Learning In Medicine, 32(2), 168–175. https://doi.org/10.1080/10401334.2019.1657869.
Sargeant, J., Mann, K., Sinclair, D., van der Vleuten, C., & Metsemakers, J. (2007). Challenges in multisource feedback: Intended and unintended outcomes. Medical Education, 41(6), 583–591. https://doi.org/10.1111/j.1365-2923.2007.02769.x.
Sargeant, J., McNaughton, E., Mercer, S., Murphy, D., Sullivan, P., & Bruce, D. A. (2011). Providing feedback: Exploring a model (emotion, content, outcomes) for facilitating multisource feedback. Medical Teacher, 33(9), 744–749. https://doi.org/10.3109/0142159x.2011.577287.
Sargeant, J., Lockyer, J., Mann, K., Holmboe, E., Silver, I., Armson, H., & Power, M. (2015). Facilitated reflective performance feedback: Developing an evidence- and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2). Academic Medicine, 90(12), 1698–1706. https://doi.org/10.1097/acm.0000000000000809.
Shankar, S., St-Onge, C., & Young, M. E. (2022). When I say… response process validity evidence. Medical Education, 56(9), 878–880.
Shaw, T., Wood, T. J., Touchie, C., Pugh, D., & Humphrey-Murto, S. M. (2021). How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover. Adv Health Sci Educ Theory Pract, 26(1), 199–214. https://doi.org/10.1007/s10459-020-09979-6.
Snyder, C. R., & Clair, M. (1976). Effects of expected and obtained grades on teacher evaluation and attribution of performance. Journal of Educational Psychology, 68, 75–82. https://doi.org/10.1037/0022-0663.68.1.75.
St-Onge, C., Young, M., Eva, K. W., & Hodges, B. (2017). Validity: One word with a plurality of meanings. Advances in Health Sciences Education, 22, 853–867.
Watling, C. J., & Lingard, L. (2012). Toward meaningful evaluation of medical trainees: The influence of participants’ perceptions of the process. Adv Health Sci Educ Theory Pract, 17(2), 183–194. https://doi.org/10.1007/s10459-010-9223-x.
Whitehead, C. R., Kuper, A., Hodges, B., & Ellaway, R. (2015). Conceptual and practical challenges in the assessment of physician competencies. Medical Teacher, 37(3), 245–251. https://doi.org/10.3109/0142159x.2014.993599.
Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching And Learning In Medicine, 15(4), 270–292. https://doi.org/10.1207/s15328015tlm1504_11.
Yeates, P., O’Neill, P., Mann, K., & Eva, K. W. (2012). Effect of exposure to good vs Poor Medical Trainee performance on attending physician ratings of subsequent Performances. Journal Of The American Medical Association, 308(21), 2226–2232. https://doi.org/10.1001/jama.2012.36515.
Yeates, P., Moreau, M., & Eva, K. (2015). Are examiners’ Judgments in OSCE-Style assessments influenced by contrast Effects? Academic Medicine, 90(7), 975–980. https://doi.org/10.1097/acm.0000000000000650.
Zhou, A. Y., & Baker, P. (2014). Confounding factors in using upward feedback to assess the quality of medical training: A systematic review. J Educ Eval Health Prof, 11, 17. https://doi.org/10.3352/jeehp.2014.11.17.
Acknowledgements
We are thankful to Shariati hospital staff for their contribution to this work.
Funding
None.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Other disclosures
None
Ethical approval
The TUMS Institutional Review Board reviewed and approved the study protocol.
Disclaimers
None.
Previous presentations
None.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Saberzadeh-Ardestani, B., Sima, A.R., Khosravi, B. et al. The impact of prior performance information on subsequent assessment: is there evidence of retaliation in an anonymous multisource assessment system?. Adv in Health Sci Educ (2023). https://doi.org/10.1007/s10459-023-10267-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10459-023-10267-2