Graphical abstract

Introduction

Background/rationale

Evaluation is an essential component of any educational process, and it provides evidence for students’ achievement of learning goals [1]. If an evaluation is accompanied by appropriate feedback, it can best improve the learner’s skills [2]. A professional evaluation of skills in healthcare settings entails evaluating both technical and non-technical skills [3]. Nurse anesthetists play an essential role in providing anesthesia care and dealing with anesthesia-related complications [4]. Traditional clinical education that is currently offered in hospitals often cannot create maximally effective learning opportunities [5]. A formative evaluation facilitates teaching and, consequently, learning [6]. Given the important role of clinical students in the safe care of patients, it is necessary to evaluate these students’ qualifications based on clear professional standards [7]. However, in order to improve services and reduce mistakes in sensitive environments such as the operating room, it is particularly important to pay attention to non-technical skills [8]. Therefore, it is now considered important to use more comprehensive methods to measure all the skills needed to provide care as part of an anesthesia team [9]. One of the new clinical evaluation methods used for assessing cognitive knowledge, as well as technical and non-technical skills, is the Anesthetic List Management Assessment Tool (ALMAT). The Royal College of Anesthetists developed the ALMAT evaluation method based on the Acute Care Assessment Tool evaluation method [10]. This evaluation model is designed to evaluate the cognitive knowledge of anesthesia residents and facilitate feedback on technical and non-technical skills [11]. ALMAT is a clinical and performance-based evaluation method, the goal of which is to bring students to competence and independence. For this reason, it is suitable for final-year students [10]. No study has yet been conducted on this evaluation method, and it has only been used as an educational-evaluation guideline at one of the British Royal Colleges for the training and evaluation of final-year anesthesia residents [11].

Objectives

We hypothesized that compared to common educational methods, the implementation of evaluation based on the ALMAT form as a formative-feedback evaluation method would improve technical and non-technical skills, and help learners achieve mastery and independence.

Methods

Ethics statement

This study was approved by the Ethics Committee of AJUMS (Ref. ID: IR.AJUMS.REC.1401.476). Informed consent was obtained from participants.

Study design

This was a quasi-experimental study, with a non-equivalent control group pre- and post-test design. It was described according to the Transparent Reporting of Evaluations with Nonrandomized Designs statement.

Setting

This study was conducted at 4 university hospitals (Golestan Hospital, Imam Hospital, Razi Hospital, Taleghani Hospital) affiliated with AJUMS during 1 academic semester (October, November, and December 2022). Weekly observations and feedback based on ALMAT (Supplement 1) to improve the technical and non-technical skills of senior students (last year of the bachelor’s degree program) continued until the students achieved mastery and independence in the targeted skills. To evaluate the effect of ALMAT on the improvement of students’ technical and non-technical skills, pre- and post-tests were used (Supplements 2, 3). Before the implementation of the intervention, both groups of students (the intervention group and the control group) took a pre-test, and after the last formative assessment using the ALMAT method in the intervention group, a post-test was conducted for both groups.

Participants

All the final-year nurse anesthesia students (n=63) of AJUMS were invited to take part in this study. Of these, 50 agreed to participate in this study and signed a written informed consent form. As the study proceeded, 5 students dropped out: 1 from the control group and 4 from the intervention group. For this reason, only the data of the students who completed the pre- and post-test were analyzed (Fig. 1).

Interventions

At first, a pre-test was conducted to evaluate the students’ technical and non-technical skills separately in both the intervention and control groups. The next step was the selection of instructors. At each of the 4 hospitals under study, a nurse anesthesia instructor with more than 5 years of experience in the clinical training of nurse anesthetists was selected. Then, all 4 instructors participated in a briefing session with the members of the research team and were acquainted with the objectives of the study, the ALMAT form, its dimensions, its items, and how to provide oral and written feedback. Each of the instructors performed 2 observations on 2 students’ performance and gave them feedback, and then the problems and ambiguities were resolved. Afterwards, the intervention started and the students in the intervention group received a formative assessment based on the ALMAT form. In this way, the performance of each student during the induction of general anesthesia was carefully examined and observed once a week by the instructor based on the ALMAT form, which included technical and non-technical items, and then verbal feedback was immediately given to the student with full details. Additionally, for more durability and impact, the feedback given to each student was carefully recorded in the ALMAT form related to that particular student, and a copy was also provided to the student. In each observation, the level of supervision required by students was determined by the instructors based on their performance (using a range of 1, 2A, 2B, 3, 4, and not applicable [N/A]), and the evaluation continued until the students reached the N/A level of mastery and independence in learning (Supplement 1). Feedback was given to the students based on the strengths and weaknesses of their performance in these areas. The frequency of observations and feedback in the intervention group varied from 6 to 10 times based on the rate and pace of their progress towards reaching the N/A level. The control group was trained in the traditional way and did not receive regular feedback until the student reached independence. In this group, the same training method and evaluation method that the trainer had used before was implemented. After the last ALMAT formative assessment was carried out in the intervention group (i.e., after the students achieved independence from their instructor’s point of view), the post-test was conducted for both groups with the same checklists used in the pre-test.

Outcomes

The outcomes of this study included students’ technical and non-technical skills before and after the ALMAT assessment and their technical and non-technical skills before and after clinical training based on the conventional method.

Data sources/measurement

The Anesthetists’ Non-Technical Skills (ANTS) standard checklist, which is designed to check the non-technical skills of anesthesia team members, was used. This instrument was developed by Flin et. [12] in 2000 in Scotland. It has 15 items in 4 areas (task management, team working, situational awareness, and decision-making). This questionnaire is scored based on a 5-point Likert scale, with 5, 4, 3, 2, and 1 representing the options “good,” “acceptable,” “moderate,” “weak,” and “not observed,” respectively (Supplement 2). ANTS is a valid and reliable standard tool that has been psychometrically analyzed. The results of factor analysis have formed the tool into 4 subscales. Cronbach’s α coefficient for the first, second, third, and fourth subscales was 0.89, 0.73, 0.88, and 0.82, respectively. The internal consistency of this instrument has been reported to be good. Furthermore, significant and strong correlations have been observed among the 4 subscales of the tool [12,13]. The second instrument included a researcher-made checklist used to measure technical skills. This checklist measured the psychomotor skills of nurse anesthesia students in anesthesia care, with 19 items in 3 subscales (before induction, during induction, and after induction). It was scored based on a 5-point Likert scale with the options “good,” “acceptable,” “moderate,” “weak,” and “not observed” represented by 5, 4, 3, 2, and 1, respectively (Supplement 3). The content validity of this checklist was confirmed by seeking the opinion of 10 faculty members of the Department of Anesthesiology of AJUMS. In order to check the reliability of the tools, the internal consistency method was used, and a Cronbach’s α coefficient of 0.88 was obtained.

Bias

In order to prevent differences in evaluators’ performance and to keep evaluations consistent, a briefing session was held for all evaluators participating in the study about the evaluation checklists and how to complete them, and an already completed form was provided to the evaluators. In order to calculate inter-rater reliability, all evaluators observed and evaluated a similar case in the presence of researchers, then they received feedback on how they made their evaluation, and the possible problems were eliminated.

Study size

Based on similar studies, the sample size calculation was conducted using G*Power ver. 3.1.9.2 (Heinrich-Heine-Universität Düsseldorf; http://www.gpower.hhu.de/), based on the independent sample Student t-test, a 2-tailed alpha of 0.05, power (1-β) of 0.8, and a large effect size of 0.8. The result showed that a sample size of 25 per group was required. Our study included 25 students in the control group and 25 students in the intervention group.

Assignment method

All final-year nurse anesthesia students who participated in the study were selected using the census method and were randomly assigned to the intervention and control groups. Randomization was done through a table of random numbers.

Blinding (masking)

No blinding was done.

Unit of analysis

The unit of analysis was the group (experimental or control).

Statistical methods

IBM SPSS ver. 25.0 (IBM Crop.) was used for data analysis. Data analysis included descriptive and inferential statistics. The latter involved comparing the mean of quantitative variables in the 2 groups using the independent t-test and paired t-test, and if the assumptions of the test were not established, its non-parametric equivalent (i.e., the Mann-Whitney test), was used. The significance level was set at P<0.05.

Results

Participants

Fifty nurse anesthesia students including 8 men (17.8%) and 37 women (82.2%) participated in this study, and there were 5 dropouts (Table 1).

Technical skills of nurse anesthesia students

The results of the study showed that the mean scores for technical skills in the post-test increased significantly compared to the pre-test (55.76±4.11 versus 87.76±2.52, P<0.0001). In the post-test, a statistically significant difference was observed between the intervention and control groups (P<0.0001). Before the intervention, a comparison of the mean scores of the technical skills and their dimensions showed no significant difference between the control and intervention groups. It can be concluded that the use of ALMAT formative-feedback assessment had a significant effect on the improvement of the technical skills of nurse anesthesia students (Tables 2, 3). Raw response data of participants for each item of the pre- and post-test for technical skills is available from Dataset 1.

Non-technical skills of nurse anesthesia students

The mean scores of non-technical skills in the post-test increased significantly compared to the pre-test (31.57±3.17 versus 2.73±66.42, P>0.0001). In the post-test, a statistically significant difference was observed between the intervention and control groups (P>0.0001). Before the intervention, a comparison of the mean scores of non-technical skills and their dimensions showed no significant difference between the control and intervention groups. It can be concluded that the use of the ALMAT formative-feedback assessment had a significant effect on the improvement of nurse anesthesia students’ non-technical skills (Tables 4, 5). Raw response data of the participants for each item of the pre- and post-test for non-technical skills is available from Dataset 2.

Discussion

Key results

This study investigated the effect of ALMAT evaluations on the improvement of technical and non-technical skills of nurse anesthesia students. Based on the findings of the present study, the intervention group showed a greater degree of improvement in technical and non-technical skills than the control group, and the hypothesis of the study was confirmed.

Interpretation

The results of the study showed that the use of ALMAT for formative assessments can accelerate mastery and independence in these skills among final-year students. Furthermore, according to the students’ scores, the use of ALMAT promoted student learning. In fact, by virtue of this assessment method, students tend to identify their strengths and weaknesses, and by trying to enhance the former and eliminate the latter, they prepare themselves to enter the clinical setting. In addition, given the fact that the students received feedback in the real environment immediately after their performance, and that these evaluations were repeated on a weekly basis, it was substantially easier for the students to move towards mastery and independence. This also contributed to the better retention of the educational points resulting from the feedback given. It should be noted that the rate of using these points in the next similar situation was higher than when these points were presented only in the classroom or at the end of an internship program. Due to the fact that the effectiveness and quality of teaching methods are usually measured by the results of students’ performance, it is necessary for managers in the field of clinical education of students to adopt appropriate methods to improve their education and evaluation. Therefore, innovative models should be used based on formative feedback such as ALMAT for the clinical training of learners [11].

Comparison with previous studies

To the best of our knowledge, based on a review of the literature, no prior studies investigated ALMAT in detail. However, this method can be compared with similar formative-feedback evaluation methods. In line with the findings of the present study, a study in India on the use of mini-Clinical Evaluation Exercise (CEX) as a method to evaluate the technical skills of anesthesia graduates showed positive results. Most of the students and professors had positive opinions about various aspects of mini-CEX, such as its easy implementation and positive educational impact [14]. Another study in Australia was conducted on formative assessments with feedback during final-year students’ internships. In this study, 86% of the students considered this method of evaluation to be useful, and they were able to achieve independence in technical skills [15]. The results of a similar study that was conducted in Iran on the effect of oral feedback on the learning outcomes of nursing students were consistent with the results of the present study. Students in the intervention group had higher scores than the control group [7]. Again, to the best of our knowledge based on a review of the literature, no studies contradicting the results of the present study were found.

Limitations

Since in the present study, evaluations were performed in a real clinical environment, it was not possible to completely homogenize the clinical cases. Further studies with a larger sample size could better show the effectiveness of this evaluation method.

Generalizability

The results of this study can be useful for clinical educators to better train and evaluate students in hospitals throughout the world.

Suggestions

The presence of nurse anesthetists is necessary during the entire anesthesia procedure in order to perform various tasks. Therefore, it is very important for them to receive accurate training and evaluations. ALMAT could be compared with other evaluation methods in terms of effectiveness.

Conclusion

Using ALMAT for the formative assessment of students in clinical settings had a significant effect on improving their technical and non-technical skills. This model is recommended to be used as a low-cost and effective method for teaching and evaluating. It can also be used along with other educational methods for a better understanding of the strengths and weaknesses of students at the bedside.

Article information

Authors’ contributions

Conceptualization: AK, MK. Data curation: AK, MK. Methodology/formal analysis/validation: AK, MK. Project administration: AK, MK. Funding acquisition: not applicable. Writing–original draft: AK, MK, VS Writing–review & editing: AK, MK, VS.

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

None.

Data availability

Data files are available from Harvard Dataverse: https://doi.org/10.7910/DVN/SZYZ45

Dataset 1. Raw response data of participants for each item of the pre- and post-test (technical skills).

jeehp-20-19-dataset1.xlsx

Dataset 2. Raw response data of the participants for each item of the pre- and post-test (non-technical skills).

jeehp-20-19-dataset2.xlsx

Acknowledgments

Supplementary materials

Fig. 1.

Flowchart of the study.

References

• 1. Immonen K, Oikarainen A, Tomietto M, Kaariainen M, Tuomikoski AM, Kaucic BM, Filej B, Riklikiene O, Flores Vizcaya-Moreno M, Perez-Canaveras RM, De Raeve P, Mikkonen K. Assessment of nursing students’ competence in clinical practice: a systematic review of reviews. Int J Nurs Stud 2019;100:103414. https://doi.org/10.1016/j.ijnurstu.2019.103414 ArticlePubMed
• 2. Bari V. Updating teaching and learning methodologies in radiology. J Coll Physicians Surg Pak 2014;24:519-522. PubMed
• 3. Khalafi A, Arman P, Manouchehrian N. Scenario-based simulation and debriefing sessions can potentially improve non-technical skills in nurse anesthetist students of Iran; a quasi-experimental study. Int J Afr Nurs Sci 2022;17:100495. https://doi.org/10.1016/j.ijans.2022.100495 Article
• 4. Harris M, Chung F. Complications of general anesthesia. Clin Plast Surg 2013;40:503-513. https://doi.org/10.1016/j.cps.2013.07.001 ArticlePubMed
• 5. Burgess A, Mellis C. Feedback and assessment for clinical placements: achieving the right balance. Adv Med Educ Pract 2015;6:373-381. https://doi.org/10.2147/AMEP.S77890 ArticlePubMedPMC
• 6. Khalili Z, Khatiban M, Ali-Mohammadi N, Molavi Vardanjani M, Tapak L, Tohidi S, Moghadarikoosha M. Comparison of two formative evaluation methods on satisfaction and learning of clinical skills in nursing students. J Nurs Educ [Internet] 2021 [cited 2023 May 2];10:42-52. Available from: http://jne.ir/article-1-1298-en.html
• 7. Hasnkhani H, Mohajjel Aghdam A, Rahmani A, Mohammadpoorfard Z. Assessing self-efficacy in clinical competence among nursing students. Educ Dev Judishapur [Internet] 2015 [cited 2023 May 2];6:107-114. Available from: https://edj.ajums.ac.ir/article_79738_4151aacf6fa3ffb7ca257cd30f4695a6.pdf?lang=en
• 8. Hagemann V, Herbstreit F, Kehren C, Chittamadathil J, Wolfertz S, Dirkmann D, Kluge A, Peters J. Does teaching non-technical skills to medical students improve those skills and simulated patient outcome? Int J Med Educ 2017;8:101-113. https://doi.org/10.5116/ijme.58c1.9f0d ArticlePubMedPMC
• 9. Khalafi A, Sharbatdar Y, Khajeali N, Haghighizadeh MH, Vaziri M. Improvement of the clinical skills of nurse anesthesia students using mini-clinical evaluation exercises in Iran: a randomized controlled study. J Educ Eval Health Prof 2023;20:12. https://doi.org/10.3352/jeehp.2023.20.12 ArticlePubMedPMC
• 10. Royal College of Anaesthetists. Curriculum for a CCT in anaesthetics. Royal College of Anaesthetists; 2010.
• 11. Royal College of Anesthesiologist. 2021 Curriculum assessment guidance: v1.4 [Internet]. Royal College of Anesthesiologist. 2022 [cited 2023 May 2]. Available from: https://www.rcoa.ac.uk/media/21746
• 12. Flin R, Patey R, Glavin R, Maran N. Anaesthetists’ non-technical skills. Br J Anaesth 2010;105:38-44. https://doi.org/10.1093/bja/aeq134 ArticlePubMed
• 13. Jirativanont T, Raksamani K, Aroonpruksakul N, Apidechakul P, Suraseranivongse S. Validity evidence of non-technical skills assessment instruments in simulated anaesthesia crisis management. Anaesth Intensive Care 2017;45:469-475. https://doi.org/10.1177/0310057X1704500410 ArticlePubMed
• 14. Kurdi MS, Hungund BR. Evaluation of mini-clinical evaluation exercise (mini-CEX) as a method for assessing clinical skills in anaesthesia postgraduate education. Indian J Anaesth 2021;65:248-252. https://doi.org/10.4103/ija.IJA_1458_20 ArticlePubMedPMC
• 15. McKenzie S, Burgess A, Mellis C. Interns reflect: the effect of formative assessment with feedback during pre-internship. Adv Med Educ Pract 2017;8:51-56. https://doi.org/10.2147/AMEP.S114480 ArticlePubMedPMC

Figure & Data

References

Citations

Citations to this article as recorded by