Hostname: page-component-848d4c4894-nr4z6 Total loading time: 0 Render date: 2024-05-04T16:09:17.425Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing the effectiveness of machine translation in the Chinese EFL writing context: A replication of Lee (2020)

Published online by Cambridge University Press:  13 March 2023

Yanxia Yang Open the ORCID record for Yanxia Yang [Opens in a new window] ,
Xiangqing Wei Open the ORCID record for Xiangqing Wei [Opens in a new window] ,
Ping Li Open the ORCID record for Ping Li [Opens in a new window]  and
Xuesong Zhai Open the ORCID record for Xuesong Zhai [Opens in a new window]
Yanxia Yang
Affiliation:
Nanjing University, China; Nanjing Agricultural University, China (yanxiayang@njau.edu.cn)
Xiangqing Wei
Affiliation:
Nanjing University, China (weixq@nju.edu.cn)
Ping Li
Affiliation:
Nanjing Agricultural University, China (liping5110@njau.edu.cn)
Xuesong Zhai
Affiliation:
Zhejiang University, China (xszhai@zju.edu.cn)
Get access Rights & Permissions [Opens in a new window]

Abstract

With the dramatic improvement in quality, machine translation has emerged as a tool widely adopted by language learners. Its use, however, has been a divisive issue in language education. We conducted an approximate replication of Lee (2020) about the impact of machine translation on EFL writing. This study used a mixed-methods approach with automatic text analyzer Coh-Metrix and human ratings, supplemented with questionnaires, interviews, and screen recordings. The findings obtained support most of the original work, suggesting that machine translation can help language learners improve their EFL writing proficiency, specifically in strengthening lexical expressions. Students generally hold positive attitudes towards machine translation, despite some skeptical views regarding the values of machine translation. Most students express a strong wish to learn how to effectively use machine translation. Machine translation literacy instruction is therefore suggested for incorporation into the curriculum for language students.

Keywords

machine translationEFL writingCoh-Metrixhuman ratingsreplication
Type
Research Article
Information
ReCALL , Volume 35 , Issue 2 , May 2023 , pp. 211 - 224
Check for updates
Copyright
© The Author(s), 2023. Published by Cambridge University Press on behalf of EUROCALL, the European Association for Computer-Assisted Language Learning

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alrajhi, A. S. (2022) Genre effect on Google Translate-assisted L2 writing output quality. ReCALL. Advance online publication. https://doi.org/10.1017/S0958344022000143 CrossRefGoogle Scholar
Bowker, L. (2020) Chinese speakers’ use of machine translation as an aid for scholarly writing in English: A review of the literature and a report on a pilot workshop on machine translation literacy. Asia Pacific Translation and Intercultural Studies, 7(3): 288–298.CrossRefGoogle Scholar
Bowker, L. & Ciro, J. B. (2019) Machine translation and global research: Towards improved machine translation literacy in the scholarly community. Bingley: Emerald Publishing. https://doi.org/10.1108/9781787567214 CrossRefGoogle Scholar
Briggs, N. (2018) Neural machine translation tools in the language learning classroom: Students’ use, perceptions, and analyses. The JALT CALL Journal, 14(1): 3–24.Google Scholar
Brudermann, C., Grosbois, M. & Sarré, C. (2021) Accuracy development in L2 writing: Exploring the potential of computer-assisted unfocused indirect corrective feedback in an online EFL course. ReCALL, 33(3): 248–264.CrossRefGoogle Scholar
Chon, Y. V., Shin, D. & Kim, G. E. (2021) Comparing L2 learners’ writing against parallel machine-translated texts: Raters’ assessment, linguistic complexity and errors. System, 96: Article 102408. https://doi.org/10.1016/j.system.2020.102408 Google Scholar
Chung, E. S. & Ahn, S. (2022) The effect of using machine translation on linguistic features in L2 writing across proficiency levels and text genres. Computer Assisted Language Learning, 35(9): 2239–2264.CrossRefGoogle Scholar
Coolican, H. (2009) Research methods and statistics in psychology (5th ed.). London: Routledge.Google Scholar
Crossley, S. A. & McNamara, D. S. (2009) Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18(2): 119–135.CrossRefGoogle Scholar
Crossley, S. A. & McNamara, D. S. (2012) Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35(2): 115–135.Google Scholar
Crossley, S. A., Salsbury, T., McNamara, D. S. & Jarvis, S. (2011) Predicting lexical proficiency in language learner texts using computational indices. Language Testing, 28(4): 561–580.CrossRefGoogle Scholar
Daems, J., Vandepitte, S., Hartsuiker, R. J. & Macken, L. (2017) Translation methods and experience: A comparative analysis of human translation and post-editing with students and professional translators. Meta, 62(2): 243–484.Google Scholar
Dancey, C. P. & Reidy, J. (2004) Statistics without maths for psychology: Using SPSS for Windows (3rd ed.). Harlow: Pearson Education Limited.Google Scholar
de la Fuente, M. J. (2002) Negotiation and oral acquisition of L2 vocabulary: The roles of input and output in the receptive and productive acquisition of words. Studies in Second Language Acquisition, 24(1): 81–112. https://doi.org/10.1017/S0272263102001043 Google Scholar
Dorst, A. G., Valdez, S. & Bouman, H. (2022) Machine translation in the multilingual classroom: How, when and why do humanities students at a Dutch university use machine translation? Translation and Translanguaging in Multilingual Contexts, 8(1): 49–66. https://doi.org/10.1075/ttmc.00080.dor Google Scholar
Ducar, C. & Schocket, D. H. (2018) Machine translation and the L2 classroom: Pedagogical solutions for making peace with Google Translate. Foreign Language Annals, 51(4): 779–795. https://doi.org/10.1111/flan.12366 Google Scholar
Enríquez Raído, V. (2014) Translation and web searching . New York: Routledge. https://doi.org/10.4324/9780203798034 CrossRefGoogle Scholar
Faul, F., Erdfelder, E., Lang, A.-G. & Buchner, A. (2007) G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2): 175–191.Google ScholarPubMed
Fredholm, K. (2019) Effects of Google Translate on lexical diversity: Vocabulary development among learners of Spanish as a foreign language. Revista Nebrija de Lingüística Aplicada a la Enseñanza de Las Lenguas, 13(26): 98–117.Google Scholar
Garcia, I. & Pena, M. I. (2011) Machine translation-assisted language learning: Writing for beginners. Computer Assisted Language Learning, 24(5): 471–487.CrossRefGoogle Scholar
Graesser, A. C. & McNamara, D. S. (2011) Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 3(2): 371–398.CrossRefGoogle ScholarPubMed
Handley, Z. (2018) Replication research in computer-assisted language learning: Replication of Neri et al. (2008) and Satar & &x04E6;zdener (2008). Language Teaching, 51(3): 417–429.CrossRefGoogle Scholar
Jacobs, H. L., Zingraf, S. A., Wormuth, D. R., Hartfiel, V. F. & Hughey, J. B. (1981) Testing ESL composition: A practical approach. Rowley: Newbury House.Google Scholar
Jin, Y. & Yang, H. (2006) The English proficiency of college and university students in China: As reflected in the CET. Language, Culture and Curriculum, 19(1): 21–36. https://doi.org/10.1080/07908310608668752 Google Scholar
Jolley, J. R. & Maimone, L. (2022) Thirty years of machine translation in language teaching and learning: A review of the literature. L2 Journal, 14(1): 26–44.CrossRefGoogle Scholar
Kelly, R. & Hou, H. (2022) Empowering learners of English as an additional language: Translanguaging with machine translation. Language and Education, 36(6): 544–559. https://doi.org/10.1080/09500782.2021.1958834 CrossRefGoogle Scholar
Kenny, D. (ed.) (2022) Machine translation for everyone: Empowering users in the age of artificial intelligence. Berlin: Language Science Press.Google Scholar
Klekovkina, V. & Denié-Higney, L. (2022) Machine translation: Friend or foe in the language classroom? L2 Journal, 14(1): 105–135. https://doi.org/10.5070/L214151723 Google Scholar
Kol, S., Schcolnik, M. & Spector-Cohen, E. (2018) Google Translate in academic writing courses. The EuroCALL Review, 26(2): 50–57. https://doi.org/10.4995/eurocall.2018.10140 CrossRefGoogle Scholar
Lam, F. S. & Pennington, M. C. (1995) The computer vs. the pen: A comparative study of word processing in a Hong Kong secondary classroom. Computer Assisted Language Learning, 8(1): 75–92.Google Scholar
Latifi, S. & Gierl, M. (2021) Automated scoring of junior and senior high essays using Coh-Metrix features: Implications for large-scale language testing. Language Testing, 38(1): 62–85. https://doi.org/10.1177/0265532220929918 CrossRefGoogle Scholar
La Torre, M. D. (1999) A web-based resource to improve translation skills. ReCALL, 11(3): 41–49.Google Scholar
Lee, S.-M. (2020) The impact of using machine translation on EFL students’ writing. Computer Assisted Language Learning, 33(3): 157–175.Google Scholar
Lee, S.-M. (2023) The effectiveness of machine translation in foreign language education: A systematic review and meta-analysis. Computer Assisted Language Learning, 36(1–2): 103–125. https://doi.org/10.1080/09588221.2021.1901745 CrossRefGoogle Scholar
Lee, S.-M. & Briggs, N. (2021) Effects of using machine translation to mediate the revision process of Korean university students’ academic writing. ReCALL, 33(1): 18–33. https://doi.org/10.1017/S0958344020000191 Google Scholar
Li, W. (2011) Moment analysis and translanguaging space: Discursive construction of identities by multilingual Chinese youth in Britain. Journal of Pragmatics, 43(5): 1222–1235. https://doi.org/10.1016/j.pragma.2010.07.035 Google Scholar
Liu, H. & Brantmeier, C. (2019) “I know English”: Self-assessment of foreign language reading and writing abilities among young Chinese learners of English. System, 80: 60–72.Google Scholar
McManus, K. (2022) Are replication studies infrequent because of negative attitudes? Insights from a survey of attitudes and practices in second language research. Studies in Second Language Acquisition, 44(5): 1410–1423. https://doi.org/10.1017/S0272263121000838 CrossRefGoogle Scholar
McNamara, D. S., Crossley, S. A. & Roscode, R. (2013) Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 45(2): 499–515.CrossRefGoogle Scholar
McNamara, D. S., Graesser, A. C., McCarthy, P. M. & Cai, Z. (2014) Automated evaluation of text and discourse with Coh-Metrix. New York: Cambridge University Press. https://doi.org/10.1017/CBO9780511894664 CrossRefGoogle Scholar
Niño, A. (2008) Evaluating the use of machine translation post-editing in the foreign language class. Computer Assisted Language Learning, 21(1): 29–49.CrossRefGoogle Scholar
Niño, A. (2020) Exploring the use of online machine translation for independent language learning. Research in Learning Technology, 28: 1–32. https://doi.org/10.25304/rlt.v28.2402 Google Scholar
O’Brien, S. & Ehrensberger-Dow, M. (2020) MT literacy—A cognitive view. Translation, Cognition & Behavior, 3(2): 145–164.CrossRefGoogle Scholar
O’Brien, S., Simard, M. & Goulet, M.-J. (2018) Machine translation and self-post-editing for academic writing support: Quality explorations. In Moorkens, J., Castilho, S., Gaspari, F. & Doherty, S. (eds.), Translation quality assessment: From principles to practice. Cham: Springer, 237–262. https://doi.org/10.1007/978-3-319-91241-7_11 CrossRefGoogle Scholar
O’Neill, E. M. (2019) Online translator, dictionary, and search engine use among L2 students. CALL-EJ, 20(1): 154–177. ’http://callej.org/journal/20-1/O’Neill2019.pdf Google Scholar
Porte, G. & McManus, K. (2019) Doing replication research in applied linguistics. New York: Routledge. https://doi.org/10.4324/9781315621395 CrossRefGoogle Scholar
Porte, G. & Richards, K. (2012) Focus article: Replication in second language writing research. Journal of Second Language Writing, 21(3): 284–293. https://doi.org/10.1016/j.jslw.2012.05.002 CrossRefGoogle Scholar
Santos, T. (1988) Professors’ reactions to the academic writing of nonnative-speaking students. TESOL Quarterly, 22(1): 69–90. https://doi.org/10.2307/3587062 CrossRefGoogle Scholar
Somers, H. (2004) Does machine translation have a role in language learning? Paper presented at UNTELE 2004: L’autonomie de l’enseignant et de l’apprenant face aux technologies de l’information et de la communication, Teacher and Learner Autonomy vis-a-vis Information Communication Technology . Universiteé de Technologie de Compiegne, 19 March.Google Scholar
Stapleton, P. & Kin, B. L. K. (2019) Assessing the accuracy and teachers’ impressions of Google Translate: A study of primary L2 writers in Hong Kong. English for Specific Purpose, 56: 18–34.Google Scholar
Tsai, S.-C. (2019) Using Google Translate in EFL drafts: A preliminary investigation. Computer Assisted Language Learning, 32(5–6): 510–526.CrossRefGoogle Scholar
Tschichold, C. (2019) CALL replication studies: Getting to grips with complexity. In Meunier, F., Van de Vyver, J., Bradley, L. & Thouësny, S. (eds.), CALL and complexity: Short papers from EUROCALL 2019. Research-publishing.net, 362–366.Google Scholar
Wiley, J., Hastings, P., Blaum, D., Jaeger, A. J. Hughes, S., Wallace, P., Griffin, T. D. & Britt, M. A. (2017) Different approaches to assessing the quality of explanations following a multiple-document inquiry activity in science. International Journal of Artificial Intelligence in Education, 27(4): 758–790.CrossRefGoogle Scholar
Yamada, M. (2019) The impact of Google Neural Machine Translation on post-editing by student translators. The Journal of Specialised Translation, 31: 87–106. https://jostrans.org/issue31/art_yamada.pdf Google Scholar
Yang, Y. & Wang, X. (2019) Modeling the intention to use machine translation for student translators: An extension of technology acceptance model. Computers & Education, 133: 116–126. https://doi.org/10.1016/j.compedu.2019.01.015 CrossRefGoogle Scholar
Zheng, Y. & Cheng, L. (2008) Test review: College English Test (CET) in China. Language Testing, 25(3): 408–417.CrossRefGoogle Scholar