Machine Translation Quality Evaluation and Post-Editing Efficiency: The Case of Abadis Translator
DOI:
https://doi.org/10.9744/kata.27.1.18-33Keywords:
Machine translation, Abadis translator, ChatGPT, Machine translation evaluation , Wilss’ matrixAbstract
Machine translation evaluation ensures that systems meet both linguistic and functional standards, providing end-users with reliable tools. This study examines the domestic machine translation system, Abadis Translator, focusing on the syntax, semantics, and pragmatics included in Wilss’ (1982) matrix. An 80-statement translation test was developed, consisting of 60 instrumental, descriptive, and argumentative statements, along with 20 idiomatic statements to evaluate pragmatic accuracy. The statements were translated using Abadis Translator and assessed using Wilss’ criteria. A Likert-scale questionnaire, with options such as incorrect, inappropriate, undesirable, correct, and appropriate, was employed. ChatGPT was used to post-edit the incorrect and inappropriate translations to enhance quality, helping to assess the reliability of both Abadis Translator and ChatGPT as an Artificial Intelligence (AI) editor. The results demonstrated that Abadis Translator exhibited strong grammatical accuracy, although it occasionally encountered subject-verb agreement issues in complex sentences. While it achieved moderate success in semantic translation, it struggled with pragmatic nuances, especially with idiomatic expressions and participle constructions. Nevertheless, post-editing with ChatGPT significantly improved overall translation quality by correcting grammatical errors and clarifying implied meanings.
Downloads
References
Abdi, H. (2016). Translation and technology: A study of Iranian freelance translators. Lambert Academic Publishing.
Abdi, H. (2021a). Considering machine translation (MT) an aid or a threat to the human translator: The case of Google Translate. Journal of Translation and Language Studies, 2(1), 19-32. https://doi.org/10.48185/jtls.v2i1.122
Abdi, H. (2021b). Examining the appropriateness of Reiss’s functionalist-oriented approach to trancism. Theory and Practice in Language Studies, 11(5), 561-567. http://dx.doi.org/10.17507/tpls.1105.15
Abdi, H. (2024). Negative analytic of the Persian translation of Atwood’s The Blind Assassin using Berman’s model of trancism. International Journal of Philology and Translation Studies, 6(1), 27-40. https://doi.org/10.55036/ufced.1434250
Aghai, M. (2024). ChatGPT vs. Google Translate: Comparative analysis of translation quality. Iranian Journal of Translation Studies, 22(85), 85-100. https://dorl.net/dor/20.1001.1.17350212.1403.22.1.9.2
Burke, A., & Maxwell, V. (2012). Lonely planet: Iran (Travel Guide). Lonely Planet Publications.
Chan, S. W. (2004). A dictionary of translation technology. Chinese University Press.
Dalaslan, D. (2015). An analysis of the English translation of Erzurum folk riddles in the light of Raymond Van den Broeck’s translation criticism model [Master’s thesis, Hacettepe University Graduate School of Social Sciences]. Hacettepe University Digital Archive. http://www.openaccess.hacettepe.edu.tr/xmlui/handle/11655/1285
Defense Advanced Research Projects Agency. (1990). Defense advanced research projects agency: Technology transition. United States Department of Defense. https://apps.dtic.mil/sti/tr/pdf/ADA434135.pdf
Defense Language Institute English Language Center. (2011). American language course - Book of idioms: Slang, special expressions, & idiomatic language. Lackland Air Force Base.
Deng, X., & Yu, Z. (2022). A systematic review of machine-translation-assisted language learning for sustainable education. Sustainability, 14(13), Article 7598. https://doi.org/10.3390/su14137598
Dorr, B., Snover, M., & Madnani, N. (2010). Machine translation evaluation. In J. McCary, C. Christianos & J. P. Olive (Eds.), Handbook of natural language processing and machine translation (pp. 801-814). Springer.
Fauziah, I. M., & Putri, R. W. P. A. (2023). Book review of Zakaryia Almahasees’s Analyzing English-Arabic Machine Translation. International Journal of Arabic-English Studies, 23(2), 443-444. https://doi.org/10.33806/ijaes.v23i2.471
Gala, J., Chitale, P., Raghavan, A. K., Gumma, V., Doddapaneni, S., Kumar, A., Nawal, J., Sujatha, A., Puduppully, R., Raghavan, V., Kumar, P., Khapra, M., & Kunchukuttan, A. (2023). IndicTrans2: Towards high-quality and accessible machine translation models for all 22 scheduled Indian languages. Transactions on Machine Learning Research, 12, 1-95. https://doi.org/10.48550/arXiv.2305.16307
Han, L. (2018, September 19). Machine translation evaluation resources and methods: A survey [Conference presentation]. Ireland Postgraduate Research Conference (IPRC), Dublin, Ireland. https://arxiv.org/abs/1605.04515
Hirschman, L., & Thompson, H. S. (1997). Overview of evaluation in speech and natural language processing. In R. Cole, J. Mariani, H. Uszkoreit, A. Zaenen, & V. Zue (Eds.), Survey of the state of the art in human language technology. Cambridge University Press
Hutchins, J., & Somers, H. (1992). An introduction to machine translation. Academic Press.
Lyu, C., Du, Z., Xu, J., Duan, Y., Wu, M., Lynn, T., Aji, A. F., Wong, D., Liu, S., & Wang, L. (2024, May 20-25). A paradigm shift: The future of machine translation lies with large language models [Conference presentation]. The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, Torino, Italy. https://aclanthology.org/2024.lrec-main.120
Makhachashvili, R., Mosiyevych, L., & Kurbatova, T. (2023). Challenges of machine translation application to teaching ESP to construction students. Social Sciences and Humanities, 3, 1-11. https://doi.org/10.55056/cs-ssh/3/04005
Mercan, H., Akgun, Y., & Odacioglu, M. C. (2024). The evolution of machine translation: A review study. International Journal of Language and Translation Studies, 4(1), 104-116.
Mirzaeian, V. R., & Oskoui, K. (2022). Investigating Iranian EFL student teachers’ attitude toward the implementation of machine translation as an ICALL tool. Journal of English Language Teaching and Learning, 14(30), 165-179. https://doi.org/10.22034/ELT.2022.52038.2496
Mistrík, J. (1997). Štylistika. Slovenské Pedagogické Nakladateľstvo.
Ntamwana, S., & Munandar, A. (2024). Translation criticism: Implementation of House’s TQA model on Sweet Hour of Prayer into Indonesian. k@ta, 26(1), 62-74. https://doi.org/10.9744/kata.26.1.62-74
Nyberg, E., Mitamura, T., & Carbonell, J. (1994, August 5). Evaluation metrics for knowledge-based translation. In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics (pp. 95–99). Kyoto, Japan. https://aclanthology.org/C94-1013.pdf
Orr, D. B., & Small, V. H. (1967). Comprehensibility of machine-aided translations of Russian scientific documents. Mechanical Translation and Computational Linguistics, 10(1-2), 1-10.
Rowling, J. K. (1997). Harry Potter and the sorcerer's stone. Scholastic Press.
Safari, M., Pourhaji, M., Fathi-Alishah, S., Sajjadi, S., & Mohammadi, M. (2023). Translating medical texts from Persian to English: Accuracy of machine translation. Archives of Advances in Bioscience, 14(E43067), 1-6. https://doi.org/10.22037/aab.v14i1.43067
Saghayan, M. H., Ebrahimi, S. F., & Bahrani, M. (2021, May 18-20). Exploring the impact of machine translation on fake news detection: A case study on Persian Tweets about COVID-19. Proceedings of the 29th Iranian Conference on Electrical Engineering (ICEE), 540–544. https://doi.org/10.1109/ICEE52715.2021.9544409
Sartipi, A., Dehghan, M., & Fatemi, A. (2023, February 1). An evaluation of Persian-English machine translation datasets with transformers. arXiv preprint arXiv:2302.00321. https://doi.org/10.48550/arXiv.2302.00321
Tan, Z., Wang, S., Yang, Z., Chen, G., Huang, X., Sun, M., & Liu, Y. (2020). Neural machine translation: A review of methods, resources, and tools. AI Open, 1, 5-21. https://doi.org/10.1016/j.aiopen.2020.11.001
Tosun, S. (2024). Machine translation: Turkish-English bilingual speakers’ accuracy detection of evidentiality and preference of MT. Cognitive Research: Principles and Implications, 9(10), 1-12. https://doi.org/10.1186/s41235-024-00535-z
van Slype, G. (1979). Critical study of methods for evaluating the quality of machine translation: Final report. Commission of the European Communities Directorate General Scientific and Technical Information and Information Management. https://aei.pitt.edu/39751/
Wang, L. (2019). Discourse-aware neural machine translation [Doctoral thesis, Dublin City University School of Computing]. Dublin City University. https://doras.dcu.ie/22903
White, J. S., O'Connell, T. A., & Carlson, L. M. (1993, March 21-24). Evaluation of machine translation [Conference presentation]. Human Language Technology Workshop, Plainsboro, New Jersey. https://aclanthology.org/H93-1040
Wilss, W. (1982). The science of translation: Problems and methods. Gunter Narr.
Yan, Y., Wang, T., Zhao, C., Huang, S., Chen, J., & Wang, M. (2023). BLEURT has universal translations: An analysis of automatic metrics by minimum risk training. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5428–5443. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.acl-long.297
Zerva, C., & Martins, A. F. T. (2024). Conformalizing machine translation evaluation. Transactions of the Association for Computational Linguistics, 12, 1460–1478. https://doi.org/10.1162/tacl_a_00711
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Hamidreza Abdi
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution License
.png)
.png)
