Evaluating the Ability of ChatGPT as an AI-Powered Chatbot: Assisting EFL Students' English Scientific Paper Writing
DOI:
https://doi.org/10.35961/salee.v7i2.2029Keywords:
ChatGPT, scientific paper, thesis, open AIAbstract
ChatGPT is an AI-powered chatbot capable of generating human-like text based on the input provided through conversations like humans in natural language. This study aims to evaluate the capability of ChatGPT as an AI-powered chatbot in assisting EFL students in English scientific paper writing, with a particular focus on undergraduate theses. This research applies a content analysis approach to examine the textual outputs generated by ChatGPT in response to academic thesis-related prompts. The researcher conducted 13 trials by inputting various thesis-related prompts into ChatGPT. The queries covered multiple components of a thesis, including the title, background (research gap, research question, and significance), literature review, methodology (design, data collection, and analysis), findings, discussion, conclusion, and references. The results indicate that ChatGPT was able to generate coherent responses for most thesis components. However, its output for the findings, discussion, and conclusion was inadequate due to the absence of actual research data and context, which are essential for producing valid results. Additionally, when given a differently formulated research objective, ChatGPT occasionally provided inconsistent or inappropriate responses. While the tool demonstrates a strong contextual understanding of text and generates relevant content, its limitations highlight potential issues related to academic integrity, including plagiarism and ethical concerns. This study emphasizes that while ChatGPT can effectively support various stages of English scientific paper writing, its outputs require careful human evaluation, verification, and refinement before being incorporated into academic work.
References
Al-Fadda, H. (2012). Difficulties in Academic Writing: From the Perspective of King Saud University Postgraduate Students. English Language Teaching, 5(3), 123–130.
Anderson, D. (2023). Master ChatGPT - Content Mastery Via Prompt for Profits: Unlock Your Content Creation Potential with Intuitive Prompts and Templates on ChatGPT. Estalontech.
Badley, G. (2009). Academic writing as shaping and re-shaping. Teaching in Higher Education, 14(2), 209–219. https://doi.org/10.1080/13562510902757294
Budjalemba, A. S., & Listyani, L. (2020). Factors Contributing To Students Difficulties In Academic Writing Class: Students Perceptions. UC Journal: ELT, Linguistics and Literature Journal, 1(2), 135–149. https://doi.org/10.24071/uc.v1i2.2966
Duin, A. H., & Pedersen, I. (2023). Augmentation Technologies and Artificial Intelligence in Technical Communication: Designing Ethical Futures. Taylor & Francis.
Fitria, T. N. (2022). Analysis of EFL Students’ Difficulties in Writing and Completing English Thesis. LLT Journal: A Journal on Language and Language Teaching, 25(1), Article 1. https://doi.org/10.24071/llt.v25i1.3607
Fitria, T. N. (2023). Artificial intelligence (AI) technology in OpenAI ChatGPT application: A review of ChatGPT in writing English essay. ELT Forum: Journal of English Language Teaching, 12(1), 44–58. https://doi.org/10.15294/elt.v12i1.64069
Fitria, T. N. (2024a). Using ChatBot-Based Artificial Intelligence (AI) for Writing an English Essay: The Ability of ChatGPT, Perplexity AI, and ChatSonic. Journal of Language Intelligence and Culture, 6(2), 103–128. https://doi.org/10.35719/jlic.v6i2.139
Fitria, T. N. (2024b). Using ChatBot-Based Artificial Intelligence (AI) for Writing an English Essay: The Ability of ChatGPT, Perplexity AI, and ChatSonic. Journal of Language Intelligence and Culture, 6(2), 103–128. https://doi.org/10.35719/jlic.v6i2.139
Fitria, T. N. (2024c). Using Google Bard as an Al-Powered Chatbot Tool for Writing English Essays. International Journal of Computer and Information System (IJCIS), 5(3), 231–245. https://doi.org/10.29040/ijcis.v5i3.171
Fitria, T. N., Simbolon, N. E., & Afdaleni, A. (2023). Chatbots as Online Chat Conversation in the Education Sector. International Journal of Computer and Information System (IJCIS), 4(3), Article 3. https://doi.org/10.29040/ijcis.v4i3.116
Gade, R. (2023). Become a Published Author Using ChatGPT: A Step-by-Step Guide to Writing and Publishing eBooks with ChatGPT. Estalontech.
Gaum, B. (2023). How to Use ChatGPT. Bernhard Gaum.
Haleem, A., Javaid, M., & Singh, R. P. (2022). An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2(4), 100089. https://doi.org/10.1016/j.tbench.2023.100089
Khalil, M., & Er, E. (2023). Will ChatGPT get you caught? Rethinking of Plagiarism Detection (arXiv:2302.04335). arXiv. https://doi.org/10.48550/arXiv.2302.04335
Kramer, C. (2023). Artificial Intelligence and the End of Humanity. MB Cooltura.
Lawrence, C. D. (2023). Hidden in White Sight: How AI Empowers and Deepens Systemic Racism. CRC Press.
Leib, R. (2023). Exoanthropology: Dialogues with AI. Punctum Books.
Luberisse, J. (2023). The Geopolitics of Artificial Intelligence: Strategic Implications of AI for Global Security. Fortis Novum Mundum.
Mardawani. (2020). Praktis Penelitian Kualitatif Teori Dasar Dan Analisis Data Dalam Perspektif Kualitatif. Deepublish.
Miles, M. B., Huberman, A. M., & Saldana, J. (2018). Qualitative Data Analysis: A Methods Sourcebook. SAGE Publications.
NeSmith, D. R. A. (2023). How ChatGPT Will Change Education. Applied Principles of Education & Learning.
Nguyen, A., Hong, Y., Dang, B., & Huang, X. (2024). Human-AI collaboration patterns in AI-assisted academic writing. Studies in Higher Education, 49(5), 847–864. https://doi.org/10.1080/03075079.2024.2323593
Nguyen, A., Ilesanmi, F., Dang, B., Vuorenmaa, E., & Järvelä, S. (2024, August 21). Hybrid Intelligence in Academic Writing: Examining Self-Regulated Learning Patterns in an AI-Assisted Writing Task [A4]. IOS Press. 6b774dfa-9bd1-4811-9696-55604aa080f0. https://oulurepo.oulu.fi/handle/10024/51567
Nkwocha, K. (2023). ChatGPT And The AI Manifesto: How AI and mankind will create a better future. Athena Publishing.
Qureshi, U. M., Tong, E. K.-M., Chen, Z. V., Deng, C., & Nandakumar, G. (2025). A Robust and Secure Framework for an AI-Assisted Academic Writing Platform. In T. Hao, J. G. Wu, X. Luo, Y. Sun, Y. Mu, S. Ge, & W. Xie (Eds.), Learning Technologies and Systems (pp. 235–249). Springer Nature. https://doi.org/10.1007/978-981-96-4407-0_20
Ryan, J., & Diorio, J. (2023). Crypto Decrypted: Debunking Myths, Understanding Breakthroughs, and Building Foundations for Digital Asset Investing. John Wiley & Sons.
Sarosa, S. (2021). Analisis Data Penelitian Kualitatif. PT Kanisius.
Setiawan, A., & Luthfiyani, U. K. (2023). Penggunaan ChatGPT Untuk Pendidikan di Era Education 4.0: Usulan Inovasi Meningkatkan Keterampilan Menulis. JURNAL PETISI (Pendidikan Teknologi Informasi), 4(1), 49–58. https://doi.org/10.36232/jurnalpetisi.v4i1.3680
Shidiq, M. (2023). The Use of Artificial Intelligence-Based Chat-GPT and its Challenges for the World of Education; From the Viewpoint of the Development of Creative Writing Skills. Proceeding of International Conference on Education, Society and Humanity, 1(1), 360–364. https://www.ejournal.unuja.ac.id/index.php/icesh/article/view/5614
Skrabut, S. (2023). 80 Ways to Use ChatGPT in the Classroom. Stan Skrabut.
Supriyadi, E. (2022). Eksplorasi Penggunaan ChatGPT dalam Penulisan Artikel Pendidikan Matematika. Papanda Journal of Mathematics and Science Research, 1(2). https://ejournal.papanda.org/index.php/pjmsr/article/view/252
Tsipursky, D. G. (2023). ChatGPT for Thought Leaders and Content Creators: Unlocking the Potential of Generative AI for Innovative and Effective Content Creation. Intentional Insights.
Wali, R. (2023). Breaking the Language Barrier: Demystifying Language Models with OpenAI: Unraveling OpenAI Frameworks for Advanced Conversational AI [ChatGPT Included]. Rayan Wali.
West, C. (2023). ChatGPT: The Future of Intelligent Conversation: How to Use ChatGPT OpenAI to Make Money, Personal Assistant, Research and Daily Life. Cea West.
Wildman, K. E. (2023). Beyond the Limits of Time (2): Advanced Techniques & Applications. Beyond The Limits Of Time, LLC.
Wiles, B. (2023). Teen Tycoon: How to Build Your Million-Dollar Empire at School. Timothy Davey.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Tira Nur Fitria
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
