Abstract
The new era of generative artificial intelligence has sparked the blossoming academic fireworks in the realm of education and information technologies. Driven by natural language processing (NLP), automated writing evaluation (AWE) tools become a ubiquitous practice in intelligent computer-assisted language learning (CALL) environments. Based on the self-set corpus of the plain text file encompassing 1524 documents from the Web of Science core collection, the current study adopts quantitative and qualitative methods and integrates bibliometric, scientometric, and meta-analytic approaches aiming to comprehensively review automated writing evaluation (AWE) over fifteen years from 2008 to 2023. Feedback literacy is the theoretical framework of automated written corrective feedback (AWCF). Through VOSviewer, this study bibliographically visualized AWE-relevant keywords, documents, authors, organizations, and regions at a macro level. Science mapping analysis (SMA), mapping knowledge domain (MKD), and author co-citation analysis (ACA) are the theoretical foundations of visualization on VOSviewer. Through Stata/SE 16 and SPSS 29, this study meta-analytically investigated moderator effects of various AWE tools, feedback types, intervention duration, target language learners, educational levels, genres of writing, regions, document types, and publication year at a micro level. It is concluded that AWE tools could facilitate writing proficiency at a statistical significance level (SMD = 0.422, p < 0.001) based on 29 experimental studies. The findings illuminate future research directions and provide heuristic implications for practitioners, researchers, and AWE technology developers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
*Studies included in the meta-analysis are marked with an asterisk
*Al-Mofti, K. (2020). The effect of using online automated feedback on Iraqi EFL learners’ writings at university level. Journal of College of Education for Women,31(3), 1–14. https://doi.org/10.36231/coeduw/vol31no3.12
*Aluthman, E. S. (2016). The effect of using automated essay evaluation on ESL undergraduate students’ writing skill. International Journal of English Linguistics,6(5), 54–67. https://doi.org/10.5539/ijel.v6n5p54
Aria, M., & Cuccurullo, C. (2017). bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics,11(4), 959–975. https://doi.org/10.1016/j.joi.2017.08.007
Bai, L., & Hu, G. (2017). In the face of fallible AWE feedback: How do students respond? Educational Psychology,37(1), 67–81. https://doi.org/10.1080/01443410.2016.1223275
*Barrot, J. S. (2021). Using automated written corrective feedback in the writing classrooms: Effects on L2 writing accuracy. Computer Assisted Language Learning,36(4), 584–607. https://doi.org/10.1080/09588221.2021.1936071
Barrot, J. S. (2023). Trends in automated writing evaluation systems research for teaching, learning, and assessment: A bibliometric analysis. Education and Information Technologies, 1–25. https://doi.org/10.1007/s10639-023-12083-y
Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Biometrika, 45(3-4), 293–315. https://doi.org/10.2307/2333180
Bennett, R. E., & Zhang, M. (2015). Validity and automated scoring. In Technology and testing (pp. 142–173). Routledge.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley & Sons Ltd, Chichester, UK. https://doi.org/10.1002/9780470743386
Brown, D., Liu, Q., & Norouzian, R. (2023). Effectiveness of written corrective feedback in developing L2 accuracy: A Bayesian meta-analysis. Language Teaching Research. https://doi.org/10.1177/13621688221147374
Bu, Y., Liu, T. Y., & Huang, W. B. (2016). MACA: a modified author co-citation analysis method combined with general descriptive metadata of citations. Scientometrics,108, 143–166. https://doi.org/10.1007/s11192-016-1959-5
Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education,43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354
Chang, C. Y., Lee, D. C., Tang, K. Y., & Hwang, G. J. (2021). Effect sizes and research directions of peer assessments: From an integrated perspective of meta-analysis and co-citation network. Computers & Education,164, 104123. https://doi.org/10.1016/j.compedu.2020.104123
*Cheng, G. (2017). The impact of online automated feedback on students’ reflective journal writing in an EFL course. The Internet and Higher Education,34, 18–27. https://doi.org/10.1016/j.iheduc.2017.04.002
*Cheng, G. (2019). Exploring the effects of automated tracking of student responses to teacher feedback in draft revision: Evidence from an undergraduate EFL writing course. Interactive Learning Environments,30(2), 353–375. https://doi.org/10.1080/10494820.2019.1655769
*Chew, C. S., Idris, N., Loh, E. F., Wu, W. C. V., Chua, Y. P., & Bimba, A. T. (2019). The effects of a theory-based summary writing tool on students’ summary writing. Journal of Computer Assisted Learning,35(3), 435–449. https://doi.org/10.1111/jcal.12349
Cochran, W. G. (1954). Some methods for strengthening the common χ2 tests. Biometrics,10(4), 417–451. https://doi.org/10.2307/3001616
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates, Publisher. https://doi.org/10.4324/9780203771587
Cook, G. (2010). Translation in language teaching: An argument for reassessment. Oxford University Press. https://doi.org/10.1093/elt/ccr007
Cooper, H. (2016). Research synthesis and meta-analysis: A step-by-step approach. SAGE. https://doi.org/10.4135/9781071878644
David, C., & Naomi, W. (2020). Teacher feedback literacy and its interplay with student feedback literacy. Teaching in Higher Education. https://doi.org/10.1080/13562517.2020.1782372
Debray, T. P. A., Moons, K. G. M., & Riley, R. D. (2018). Detecting small-study effects and funnel plot asymmetry in meta-analysis of survival data: A comparison of new and existing tests. Research Synthesis. Methods,9(1), 41–50. https://doi.org/10.1002/jrsm.1266
Ding, L., & Zou, D. (2024). Automated writing evaluation systems: A systematic review of Grammarly, Pigai, and Criterion with a perspective on future directions in the age of generative artificial intelligence. Education and Information Technologies. https://doi.org/10.1007/s10639-023-12402-3
Dong, Z., Gao, Y., & Schunn, C. D. (2023). Assessing students’ peer feedback literacy in writing: Scale development and validation. Assessment and Evaluation in Higher Education, 1–16. https://doi.org/10.1080/02602938.2023.2175781
Duran, R. P., Eisenhart, M. A., Erickson, F. D., Grant, C. A., Green, J. L., Hedges, L. V., & Schneider, B. L. (2006). Standards for reporting on empirical social science research in AERA publications: American Educational Research Association. Educational Researcher,35(6), 33–40. https://doi.org/10.3102/0013189X035006033
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ,315(7109), 629–634.
Ellis, R. (2009). A typology of written corrective feedback types. ELT Journal,63, 97–107. https://doi.org/10.1093/elt/ccn023
Ellis, R., Sheen, Y., Murakami, M., & Takashima, H. (2008). The effects of focused and unfocused written corrective feedback in an English as a foreign language context. System,36, 353–371. https://doi.org/10.1016/j.system.2008.02.001
Elola, I., & Oskoz, A. (2010). Collaborative writing: fostering foreign language and writing conventions development. Language Learning & Technology, 14)3, 51–71. 10125/44226
Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: Insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education,20, 55. https://doi.org/10.1186/s41239-023-00425-2
*Fan, N. (2023). Exploring the effects of automated written corrective feedback on EFL students’ writing quality: A mixed- methods study. SAGE Open, 13(2), 21582440231181296. https://doi.org/10.1177/21582440231181296
Fleckenstein, J., Liebenow, L. W., & Meyer, J. (2023). Automated feedback and writing: a multi-level meta-analysis of effects on students’ performance. Frontiers in Artificial Intelligence,6, 1162454. https://doi.org/10.3389/frai.2023.1162454
*Frost, K. L. (2009). The effects of automated essay scoring as a high school classroom intervention. [Unpublished doctoral dissertation]. University of Nevada.
Fu, Q. K., Zou, D., Xie, H., & Cheng, G. (2022). A review of AWE feedback: types, learning outcomes, and implications. Computer Assisted Language Learning, 1–43. https://doi.org/10.1080/09588221.2022.2033787
*Gao, J., & Ma, S. (2019). The effect of two forms of computer-automated metalinguistic corrective feedback. Language Learning & Technology, 23(2), 65–83. https://doi.org/10125/44683
*Gao, J., & Ma, S. (2020). Instructor feedback on free writing and automated corrective feedback in drills: Intensity and efficacy. Language Teaching Research. https://doi.org/10.1177/1362168820915337
Grimes, D., & Warschauer, M. (2010). Utility in a fallible tool: A multi-site case study of automated writing evaluation. Journal of Technology, Learning, and Assessment, 8(6). Retrieved [date] from http://www.jtla.org. Accessed 9 March 2024.
*Han, T., & Sari, E. (2022). An investigation on the use of automated feedback in Turkish EFL students' writing classes. Computer Assisted Language Learning, 1–25. https://doi.org/10.1080/09588221.2022.2067179
*Han, Y., Zhao, S., & Ng, L. (2021). How technology tools impact writing performance, lexical complexity, and perceived self-regulated learning strategies in EFL academic writing: A comparative study. Frontiers in Psychology,12, 1–18. https://doi.org/10.3389/fpsyg.2021.752793
Haque, M., Kumar, V. V., Singh, P., Goyal, A. A., Upreti, K., & Verma, A. (2023). A systematic meta-analysis of blockchain technology for educational sector and its advancements towards education 4.0. Education and Information Technologies, 1–27. https://doi.org/10.1007/s10639-023-11744-2
Harrer, M., Cuijpers, P., Furukawa, T., & Ebert, D. (2021). Doing meta-analysis with R: A hands-on guide. Chapman and Hall/CRC. https://doi.org/10.1201/9781003107347
*Hassanzadeh, M., & Fotoohnejad, S. (2021). Implementing an automated feedback program for a foreign language writing course: A learner-centric study: Implementing an AWE tool in a L2 class. Journal of Computer Assisted Learning,37(5), 1494–1507. https://doi.org/10.1111/jcal.12587
Hedges, L. V., & Olkin, I. (2014). Statistical methods for meta-analysis. Academic press
Heift, T., & Hegelheimer, V. (2017). Computer-assisted corrective feedback and language learning. Corrective Feedback in Second Language Teaching and Learning, 51–65. https://doi.org/10.4324/9781315621432-5
Heigham, J., & Croker, R. A. (Eds.). (2009). Qualitative research in applied linguistics: A practical introduction. Palgrave Macmillan. https://doi.org/10.1057/9780230239517
Higgins, J. P. T., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions, John Wiley & Sons. https://doi.org/10.1002/9780470712184. ISBN 1119964792.
Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine,21(11), 1539–1558. https://doi.org/10.1002/sim.1186
Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ,327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557
Ho, R. (2013). Handbook of univariate and multivariate data analysis with IBM SPSS (2nd ed.). Chapman and Hall/CRC. https://doi.org/10.1201/b15605
Hockly, N. (2019). Automated writing evaluation. ELT Journal,73(1), 82–88. https://doi.org/10.1093/elt/ccy044
Hou, J. (2017). Exploration into the evolution and historical roots of citation analysis by referenced publication year spectroscopy. Scientometrics,110, 1437–1452. https://doi.org/10.1007/s11192-016-2206-9
*Huang, S., & Renandya, W. A. (2020). Exploring the integration of automated feedback among lower-proficiency EFL learners. Innovation in Language Learning and Teaching,14(1), 15–26. https://doi.org/10.1080/1750122
Huawei, S., & Aryadoust, V. (2023). A systematic review of automated writing evaluation systems. Education and Information Technologies,28(1), 771–795. https://doi.org/10.1007/s10639-022-11200-7
Hunter, J. E., & Schmidt, F. L. (2014). Methods of meta-analysis: Correcting error and bias in research findings. Sage Publications.
Hyland, K. (1990). A genre description of the argumentative essay. RELC Journal,21(1), 66–78. https://doi.org/10.1177/003368829002100105
Hyland, K., & Hyland, F. (2019). Contexts and issues in feedback on L2 writing. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 1–22). Cambridge University Press. https://doi.org/10.1017/9781108635547.003
Jacobs, H. L. (1981). Testing ESL composition: A practical approach (p. 01969). Newbury House Publishers Inc.
Johri, P., Khatri, S. K., Al-Taani, A. T., Sabharwal, M., Suvanov, S., & Kumar, A. (2021). Natural language processing: history, evolution, application, and future work. In A. Abraham, O. Castillo, & D. Virmani (Eds.), Proceedings of 3rd international conference on computing informatics and networks. Lecture notes in networks and systems (Vol. 167). Springer. https://doi.org/10.1007/978-981-15-9712-1_31
*Karim, K., & Nassaji, H. (2020). The revision and transfer effects of direct and indirect comprehensive corrective feedback on ESL students’ writing. Language Teaching Research,24, 519–539. https://doi.org/10.1177/1362168818802469
Knapp, G., & Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine,22(17), 2693–2710. https://doi.org/10.1002/sim.1482
Koltovskaia, S. (2020). Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing,44, 100450. https://doi.org/10.1016/j.asw.2020.100450
Krishna, K., Song, Y., Karpinska, M., Wieting, J. & Iyyer, M. (2023). Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense. [preprint in ArXiv], abs/2303.13408. https://doi.org/10.48550/arXiv.2303.13408
Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. Tesol Quarterly, 49(4), 757–786. https://doi.org/10.1002/tesq.194
*Lachner, A., Burkhart, C., & Nückles, M. (2017). Mind the gap! Automated concept map feedback supports students in writing cohesive explanations. Journal of Experimental Psychology: Applied. https://doi.org/10.1037/xap000011. Advance Online Publication.
*Lai, Y. (2010). Which do students prefer to evaluate their essays: Peers or computer program. British Journal of Educational Technology,41(3), 432–454. https://doi.org/10.1111/j.1467-8535.2009.00959.x
Laird, A. R., Lancaster, J. J., & Fox, P. T. (2005). BrainMap: The social evolution of a human brain mapping database. Neuroinformatics, 3(1), 065–078. https://doi.org/10.1385/NI:3:1:065
Larivière, S., Bayrak, Ş, de Wael, R. V., Benkarim, O., Herholz, P., Rodriguez-Cruces, R., Paquola, C., Hong, S., Misic, B., Evans, A., Valk, S., & Bernhardt, B. C. (2023). BrainStat: A toolbox for brain-wide statistics and multimodal feature associations. NeuroImage, 266, 119807. https://doi.org/10.1016/j.neuroimage.2022.119807
Lauriola, I., Lavelli, A., & Aiolli, F. (2022). An introduction to deep learning in natural language processing: Models, techniques, and tools. Neurocomputing,470, 443–456. https://doi.org/10.1016/j.neucom.2021.05.103
*Lee, C., Wong, K. C. K., Cheung, W. K., & Lee, F. S. L. (2009). Web-based essay critiquing system and EFL students’ writing: A quantitative and qualitative investigation. Computer Assisted Language Learning,22(1), 57. https://doi.org/10.1080/09588220802613807
Li, R. (2023). Still a fallible tool? Revisiting effects of automated writing evaluation from activity theory perspective. British Journal of Educational Technology,54(3), 773–789. https://doi.org/10.1111/bjet.13294
Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing,27, 1–18. https://doi.org/10.1016/j.jslw.2014.10.004
Li, S., & Roshan, S. (2019). The associations between working memory and the effects of four different types of written corrective feedback. Journal of Second Language Writing,45, 1–15. https://doi.org/10.1016/j.jslw.2019.03.003
Liao, H. C. (2016). Using automated writing evaluation to reduce grammar errors in writing. Elt Journal,70(3), 308–319. https://doi.org/10.1093/elt/ccv058
Lin, J. J., & Lin, H. (2019). Mobile-assisted ESL/EFL vocabulary learning: A systematic review and meta-analysis. Computer Assisted Language Learning,32(8), 878–919. https://doi.org/10.1080/09588221.2018.1541359
Liu, Q., & Brown, D. (2015). Methodological synthesis of research on the effectiveness of corrective feedback in L2 writing. Journal of Second Language Writing, 30, 66–81. https://doi.org/10.1016/j.jslw.2015.08.011
*Liu, C., Hou, J., Tu, Y., Wang, Y., & Hwang, G. (2021). Incorporating a reflective thinking promoting mechanism into artificial intelligence-supported English writing environments. Interactive Learning Environments, 1–19. https://doi.org/10.1080/10494820.2021.2012812
Lozano, S., Calzada-Infante, L., Adenso-Díaz, B., & García, S. (2019). Complex network analysis of keywords co-occurrence in the recent efficiency analysis literature. Scientometrics,120, 609–629. https://doi.org/10.1007/s11192-019-03132-w
Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4), 474–496. https://doi.org/10.1075/ijcl.15.4.02lu
Lu, X. (2011). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly, 45(1), 36–62. https://doi.org/10.5054/tq.2011.240859
*Ma, K. (2013). Improving EFL graduate students’ proficiency in writing through an online automated essay assessing system. English Language Teaching, 6(7), 158–167. https://doi.org/10.5539/elt.v6n7p158
Masoumi, S., & Shahraz, S. (2022). Meta-analysis using Python: A hands-on tutorial. BMC Medical Research Methodology,22(1), 193. https://doi.org/10.1186/s12874-022-01673-y
Matsuura, K. (2023). Bayesian Statistical Modeling with Stan, R, and Python. Springer Nature.
McCain, K. W. (1990). Mapping authors in intellectual space: A technical overview. Journal of the American Society for Information Science,41(6), 433–443.
Mohsen, M. A. (2022). Computer-mediated corrective feedback to improve L2 writing skills: A meta-analysis. Journal of Educational Computing Research,60(5), 1253–1276. https://doi.org/10.1177/07356331211064066
Mohsen, M. A., Mahdi, H. S., & Alkhammash, R. (2023). Multimedia glosses and their impact on second language vocabulary acquisition: Insights from a meta-analysis and document co-citation analysis. Innovation in Language Learning and Teaching,1–16. https://doi.org/10.1080/17501229.2023.2236084
Molloy, E., Boud, D., & Henderson, M. (2020). Developing a learning-centered framework for feedback literacy. Assessment & Evaluation in Higher Education,45(4), 527–540. https://doi.org/10.1080/02602938.2019.1667955
Morphy, P., & Graham, S. (2012). Word processing programs and weaker writers/readers: A meta-analysis of research findings. Reading and Writing,25(3), 641–678. https://doi.org/10.1007/s11145-010-9292-5
Mulchenko, Z. M. (1971). Measurement of science. study of the development of science as an information process. Proceedings of the National Academy of Sciences of the United States of America, 405(4), 210.
Ngo, T. T. N., Chen, H. H. J., and Lai, K. K. W. (2022). The effectiveness of automated writing evaluation in EFL/ESL writing: a three-level meta-analysis. Interactive Learning Environments, 1–18. doi: 10.1080/10494820.2022.2096642.
Nunes, A., Cordeiro, C., Limpo, T., & Castro, S. L. (2022). Effectiveness of automated writing evaluation systems in school settings: A systematic review of studies from 2000 to 2020. Journal of Computer Assisted Learning,38(2), 599–620. https://doi.org/10.1111/jcal.12635
O’Brien, S., Simard, M., & Goulet, M. J. (2018). Machine translation and self-post-editing for academic writing support: Quality explorations. In J. Moorkens, S. Castilho, F. Gaspari, & S. Doherty (Eds.), Translation quality assessment. Machine translation: technologies and applications (Vol. 1). Springer. https://doi.org/10.1007/978-3-319-91241-7_11
Palmer, T.M., & Sterne, J.A. (2016). Meta-analysis in stata: An updated collection from the stata journal (2nd ed.). StataCorp LP. ISBN 978–1–59718–147–1.
Pekrun, R., Cusack, A., Murayama, K., Elliot, A. J., & Thomas, K. (2014). The power of anticipated feedback: Effects on students’ achievement goals and achievement emotions. Learning and Instruction,29, 115–124. https://doi.org/10.1016/j.learninstruc.2013.09.002
Pritchard, A. (1969). Statistical bibliography or bibliometrics. Journal of Documentation,25(4), 348–349.
Rad, H. S., Alipour, R., & Jafarpour, A. (2023). Using artificial intelligence to foster students’ writing feedback literacy, engagement, and outcome: a case of Wordtune application. Interactive Learning Environments,1–21. https://doi.org/10.1080/10494820.2023.2208170
Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning,31(7), 653–674. https://doi.org/10.1080/09588221.2018.1428994
Ranalli, J., Feng, H. H., & Chukharev-Hudilainen, E. (2018). Exploring the potential of process-tracing technologies to support assessment for learning of L2 writing. Assessing Writing,36, 77–89. https://doi.org/10.1016/j.asw.2018.03.007
*Rassaei, E. (2023). The interplay between corrective feedback timing and foreign language anxiety in L2 development. Language Teaching Research. https://doi.org/10.1177/13621688231195141
Rich, C. S. (2012). The impact of online automated writing evaluation: A case study from Dalian. Chinese Journal of Applied Linguistics,35(1), 63–79. https://doi.org/10.1515/cjal-2012-0006
Richards, J. C., & Schmidt, R. W. (2010). Longman dictionary of language teaching and applied linguistics (4th ed.). Pearson Education Limited. https://doi.org/10.4324/9781315833835
Salo, T., Yarkoni, T., Nichols, T. E., Poline, J. B., Bilgel, M., Bottenhorn, K. L., Eickhoff, S. B., Jarecka, D., Kent, J. D., Kimbler, A., Nielson, D. M., Oudyk, K. M., Peraza, J. A., Pérez, A., Reeders, P. C., Yanes, J. A., & Laird, A. R. (2023). NiMARE: Neuroimaging meta-analysis research environment. NeuroLibre, 1, 7(1). https://doi.org/10.52294/001c.87681
Sen, S., & Yildirim, I. (2022). A Tutorial on How to Conduct Meta-Analysis with IBM SPSS Statistics. Psych,4(4), 640–667. https://doi.org/10.3390/psych4040049
Sherafati, N., Largani, F. M., & Amini, S. (2020). Exploring the effect of computer-mediated teacher feedback on the writing achievement of Iranian EFL learners: Does motivation count? Education and Information Technologies,25, 4591–4613. https://doi.org/10.1007/s10639-020-10177-5
Shintani, N. (2017). The effects of the timing of isolated FFI on the explicit knowledge and written accuracy of learners with different prior knowledge of the linguistic target. Studies in Second Language Acquisition,39(1), 129–166. https://doi.org/10.1017/S0272263116000127
*Shintani, N., & Aubrey, S. (2016). The effectiveness of synchronous and asynchronous written corrective feedback on grammatical accuracy in a computer-mediated environment. The Modern Language Journal,100(1), 296–319. https://doi.org/10.1111/modl.12317
Shintani, N., Ellis, R., & Suzuki, W. (2014). Effects of written feedback and revision on learners’ accuracy in using two English grammatical structures. Language Learning,64(1), 103–131. https://doi.org/10.1111/lang.12029
Sidik, K., & Jonkman, J. N. (2002). A simple confidence interval for meta-analysis. Statistics in Medicine,21(21), 3153–3159. https://doi.org/10.1002/sim.1262
Soava, G., Mehedintu, A., Sterpu, M., & Raduteanu, M. (2018). Impact of renewable energy consumption on economic growth: Evidence from European Union countries. Technological and Economic Development of Economy,24(3), 914–932. https://doi.org/10.3846/tede.2018.1426
Stamatatos, E. (2009). A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology,60(3), 538–556. https://doi.org/10.1002/asi.21001
Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing,19, 51–65. https://doi.org/10.1016/j.asw.2013.11.007
*Sun, B., & Fan, T. (2022). The effects of an AWE-aided assessment approach on business English writing performance and writing anxiety: A contextual consideration. Studies in Educational Evaluation, 72, 1–10. https://doi.org/10.1016/j.stueduc.2021.101123
*Sun, C. (2012). The impact of online automated writing evaluation: A case study from Dalian. Chinese Journal of Applied Linguistics,35(1), 63–79. https://doi.org/10.1515/cjal-2012-0006
Tang, J., & Rich, C. S. (2017). Automated writing evaluation in an EFL setting: Lessons from China. Jalt Call Journal, 13(2), 117–146. https://doi.org/10.29140/jaltcall.v13n2.215
Teakel, S., Linden, K., van der Ploeg, N., and Roman, N. (2023). Embedding equity: online tutor support to provide effective feedforward on assessments. Assessment and Evaluation in Higher Education, 1–14. doi: 10.1080/02602938.2023.2232955.
Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learning,46(2), 327–369. https://doi.org/10.1111/j.1467-1770.1996.tb01238.x
Valverde, S., Cabezas, M., Roura, E., González-Villà, S., Pareto, D., Vilanova, J. C., et al. (2017). Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach. NeuroImage, 155, 159–168. https://doi.org/10.1016/j.neuroimage.2017.04.034
Van Eck, N., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics,84(2), 523–538. https://doi.org/10.1007/s1119200901463
Van Eck, N. J., & Waltman, L. (2017). Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics,111, 1053–1070. https://doi.org/10.1007/s11192-017-2300-7
Van Eck, N.J., & Waltman, L. (2023). VOSviewer Manual: Manual for VOSviewer version 1.6.20. Universiteit Leiden.
Venables, L., & Fairclough, S. H. (2009). The influence of performance feedback on goal-setting and mental effort regulation. Motivation and Emotion,33(1), 63–74. https://doi.org/10.1007/s11031-008-9116-y
Viechtbauer, W. (2005). Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics,30(3), 261–293. https://doi.org/10.3102/10769986030003261
*Waer, H. (2021). The effect of integrating automated writing evaluation on EFL writing apprehension and grammatical knowledge. Innovation in Language Learning and Teaching, 1–25. https://doi.org/10.1080/17501229.2021.1914062
*Wang, J. (2019). A comparative study on the washback effects of teacher feedback plus intelligent feedback versus teacher feedback on English writing teaching in higher vocational college. Theory and Practice in Language Studies, 9(12), 1555–1561. https://doi.org/10.17507/tpls.0912.12
*Wang, Y.-J., Shang, H.-F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning,26(3), 234–257. https://doi.org/10.1080/09588221.2012.655300
*Wilson, J., & Roscoe, R. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research,58(1), 87–125. https://doi.org/10.1177/0735633119830764
Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., & Wager, T. D. (2011). Large-scale automated synthesis of human functional neuroimaging data. Nature Methods, 8, 665–670. https://doi.org/10.1038/nmeth.1635
Yu, S., Xu, H., Jiang, L., & Chan, I. K. I. (2020). Understanding Macau novice secondary teachers’ beliefs and practices of EFL writing instruction: A complexity theory perspective. Journal of Second Language Writing, 48, 100728. https://doi.org/10.1016/j.jslw.2020.100728
Yu, S., & Liu, C. (2021). Improving student feedback literacy in academic writing: An evidence-based framework. Assessing Writing,48, 100525. https://doi.org/10.1016/j.asw.2021.100525
Yu, Q., Yu, K., & Li, B. (2023). Can gamification enhance online learning? Evidence from a meta-analysis. Education and Information Technologies, 1–29. https://doi.org/10.1007/s10639-023-11977-1
Zhai, N., & Ma, X. (2021). Automated writing evaluation (AWE) feedback: A systematic investigation of college students’ acceptance. Computer Assisted Language Learning. https://doi.org/10.1080/09588221.2021.1897019
Zhai, N., & Ma, X. (2023). The effectiveness of automated writing evaluation on writing quality: A meta-analysis. Journal of Educational Computing Research,61(4), 875–900. https://doi.org/10.1177/07356331221127300
*Zhang, Q. (2014). An experimental research on applying automated essay scoring system to college English writing course. International Journal of English Language Teaching,1(2), 35–41. https://doi.org/10.5430/ijelt.v1n2p35
Zhang, Q. (2024). The affective and performative effects of ludicization on language learning: An integrated model related to technology acceptance and multidimensional motivation. Education and Information Technologies, 1–34,. https://doi.org/10.1007/s10639-024-12538-w
Zhang, Z. V. (2020). Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Assessing Writing,43, 100439.
Zhang, Z., & Hyland, K. (2023). The role of digital literacy in student engagement with automated writing evaluation (AWE) feedback on second language writing. Computer Assisted Language Learning, 1–26. https://doi.org/10.1080/09588221.2023.2256815
Acknowledgements
The author would like to extend her heartfelt gratitude to anonymous reviewers and editors. The author sincerely appreciates the constructive and thorough feedback provided by editors and reviewers. The author sincerely appreciates Doctor Zhang Qi, Professor Liu Linjun, Professor Zhu Erqian, Professor Xu Hongchen, beloved family, kind friends, and Beijing Language and Culture University. The process of writing academic articles is not only a self-actualization, but also a panacea.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Contributions
Yi Xue: Conceptualization, Methodology, Software, Validation, Formal Analysis, Investigation, Resources, Data Curation, Data collection, Data analysis, Investigation, Writing-Original Draft Preparation, Writing-Review and Editing, Visualization, Funding Acquisition.
Corresponding author
Ethics declarations
Declarations
The author declares no conflicts of interest that are relevant to the content of this article. The study complies with Ethical Standards. All data and materials support our published claims and comply with field standards.
Conflict of interest
The author has no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Xue, Y. Towards automated writing evaluation: A comprehensive review with bibliometric, scientometric, and meta-analytic approaches. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12596-0
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10639-024-12596-0