(Department of Computer Science, Faculty of Engineering & Science, Dili Institute of Technology, Dili, Timor-Leste)(2) * Azhari Azhari
(Department of Computer Science and Electronics, Faculty of Natural Science Universitas Gadjah Mada, Yogyakarta, Indonesia)(3) Nur Rohkman
(Department of Computer Science and Electronics, Faculty of Natural Science Universitas Gadjah Mada, Yogyakarta, Indonesia)*corresponding author
AbstractQuestion classification (QC) is critical in an educational question-answering (QA) system. However, most existing models suffer from limited semantic accuracy, particularly when dealing with complex or ambiguous education queries. The problem lies in their reliance on surface-level features, such as keyword matching, which hampers their ability to capture deeper syntactic and semantic relationship in question. This results in misclassification and generic responses that fail to address the specific intent of prospective students. This study addresses, this gap by integrating semantic dependency parsing into Semantic-BERT (S-BERT) and Semantic-FastText (S-FastText) to enhance question classification performance. Semantic dependency parsing is applied to structure the semantics of interrogative sentences before classification processing by BERT and FastText. A dataset of 2,173 educational questions covering five question classes (5W1H) is used for training and validation. The model evaluation uses a confusion matrix and K-Fold cross-validation, ensuring robust performance assessment. Experimental results show that both models achieve 100% accuracy, precision, and recall in classifying question sentences, demonstrating their effectiveness in educational question classification. These findings contribute to the development of intelligent educational assistants, paving the way for more efficient and accurate automated question-answering systems in academic environments.
KeywordsQuestion classification; Semantic parsing; S-Bert; S-FastText; K-fold cross-validation
|
DOIhttps://doi.org/10.26555/ijain.v11i2.1955 |
Article metricsAbstract views : 1666 | PDF views : 91 |
Cite |
Full Text Download
|
References
[1] H. Gweon and M. Schonlau, “Automated Classification For Open-Ended Questions With BERT,” J. ofSurvey Stat. Methodol., vol. 12, pp. 493–504, 2024, doi: 10.1093/jssam/smad015.
[2] O. Galal, A. H. Abdel-Gawad, and M. Farouk, “Rethinking of BERT Sentence Embedding for Text Classification,” Neural Comput. Appl., vol. 36, no. 32, pp. 20245–20258, 2024, doi: 10.1007/s00521-024-10212-3.
[3] X. Fu et al., “SS-BERT : A Semantic Information Selecting Approach for Open-Domain Question Answering,” MDPI Electron., pp. 1–14, 2023, doi: 10.3390/electronics12071692.
[4] M. Khuntia and D. Gupta, “Indian News Headlines Classification using Word Embedding Techniques and LSTM Model,” Procedia Comput. Sci., vol. 218, pp. 899–907, 2022, doi: 10.1016/j.procs.2023.01.070.
[5] S. Al Faraby, A. Romadhony, and Adiwijaya, “Analysis of LLMs for educational question classification and generation,” Comput. Educ. Artif. Intell., vol. 7, p. 100298, Dec. 2024, doi: 10.1016/j.caeai.2024.100298.
[6] A. Rita Gonçalves, D. Costa Pinto, H. Gonzalez-Jimenez, M. Dalmoro, and A. S. Mattila, “Me, Myself, and My AI: How artificial intelligence classification failures threaten consumers’ self-expression,” J. Bus. Res., vol. 186, p. 114974, Jan. 2025, doi: 10.1016/j.jbusres.2024.114974.
[7] M. S. Salim and S. I. Hossain, “An Applied Statistics dataset for human vs AI-generated answer classification,” Data Br., vol. 54, p. 110240, 2024, doi: 10.1016/j.dib.2024.110240.
[8] Y. Chae and T. Davidson, “Large Language Models for Text Classification: From Zero-Shot Learning to Instruction-Tuning,” Sociol. Methods Res., p. 29, Apr. 2025, doi: 10.1177/00491241251325243.
[9] M. O. Gani et al., “Towards Enhanced Assessment Question Classification: a Study using Machine Learning, Deep Learning, and Generative AI,” Conn. Sci., vol. 37, no. 1, 2025, doi: 10.1080/09540091.2024.2445249.
[10] H. Sharma, R. Mathur, T. Chintala, S. Dhanalakshmi, and R. Senthil, “An Effective Deep Learning Pipeline for Improved Question Classification into Bloom’s Taxonomy’s Domains,” Educ. Inf. Technol., vol. 28, no. 5, pp. 5105–5145, 2023, doi: 10.1007/s10639-022-11356-2.
[11] D. Han, T. Tohti, and A. Hamdulla, “Attention-Based Transformer-BiGRU for Question Classification,” Inf., vol. 13, no. 5, 2022, doi: 10.3390/info13050214.
[12] S. Aburass, O. Dorgham, and M. A. Rumman, “An Ensemble Approach to Question Classification: Integrating Electra Transformer, GloVe, and LSTM,” Int. J. Adv. Comput. Sci. Appl., vol. 15, no. 1, pp. 507–514, 2024, doi: 10.14569/IJACSA.2024.0150148.
[13] L. Escouflaire, A. Descampe, and C. Fairon, “Automated text classification of opinion vs. news French press articles. A comparison of transformer and feature-based approaches,” Lang. Commun., vol. 99, pp. 129–140, 2024, doi: 10.1016/j.langcom.2024.09.004.
[14] S. Al Faraby, Romadhony, and A. Romadhony, “Educational Question Classification with Pre-trained Language Models,” IEEE Xplore Seventh Int. Conf. Informatics Comput., no. 156, pp. 1–6, 2022, doi: 10.1109/ICIC56845.2022.10006957.
[15] G. Xiao, E. Chow, H. Chen, J. Mo, J. Gui, and X. Gong, “Chinese Question Classification in the Low Domain,” Fourteenth IEEE Int. Conf. E-bus. Eng., pp. 214–219, 2017, doi: 10.1109/ICEBE.2017.41.
[16] I. V. C. Motta, N. Vuillerme, H. H. Pham, and F. A. P. de Figueiredo, “Machine Learning Techniques for Coffee Classification: a Comprehensive Review of Scientific Research,” Artif. Intell. Rev., vol. 58, no. 1, 2025, doi: 10.1007/s10462-024-11004-w.
[17] K. Fujiwara, “Knowledge Distillation with Resampling for Imbalanced data Classification: Enhancing Predictive Performance and Explainability Stability,” Results Eng., vol. 24, no. November, p. 103406, 2024, doi: 10.1016/j.rineng.2024.103406.
[18] I. Martinsen, D. Wade, B. Ricaud, and F. Godtliebsen, “The 3-billion Fossil Question: How to Automate Classification of Microfossils,” Artif. Intell. Geosci., vol. 5, no. April, p. 100080, 2024, doi: 10.1016/j.aiig.2024.100080.
[19] S. Rizou et al., “Efficient intent classification and entity recognition for university administrative services employing deep learning models,” Intell. Syst. with Appl., vol. 19, p. 200247, Sep. 2023, doi: 10.1016/j.iswa.2023.200247.
[20] H. Sobhanam and J. Prakash, “Analysis of fine tuning the hyper parameters in RoBERTa model using genetic algorithm for text classification,” Int. J. Inf. Technol., vol. 15, no. 7, pp. 3669–3677, 2023, doi: 10.1007/s41870-023-01395-4.
[21] D. Sarkar, Text Analytics with Python. Berkeley, CA: Apress, p. 674, 2019, doi: 10.1007/978-1-4842-4354-1.
[22] N. Indurkhya and F. J. Damerau, Handbook of Natural Language Processing, 2nd Editio. United States of America: Chapman and Hall/CRC, 2010, doi: 10.1201/9781420085938.
[23] B. Steven, K. Ewan, and L. Edward, Natural Language Processing with Python. United States of America: O’Reilly Media, Inc, p. 504, 2009. [Online]. Available at: https://www.google.co.id/books/edition/Natural_Language_Processing_with_Python/KGIbfiiP1i4C?hl=en&gbpv=0.
[24] B. Srinivasa-desikan, Natural Language Processing and Computational Linguistics, vol. 6, no. 11. UK: Packt Publishing, pp. 1- 388, 2018. [Online]. Available at: https://www.arcjournals.org/pdfs/ijsell/v6-i11/2.pdf.
[25] M. Swamynathan, Mastering Machine Learning with Python in Six Steps, 2nd ed., vol. 19, no. 2. Berkeley, CA: Apress, 2019, doi: 10.1007/978-1-4842-4947-5.
[26] A. A. Khan, “Balanced Split: A new train-test data splitting strategy for imbalanced datasets,” arXiv, pp. 1–5, 2022, [Online]. Available at: https://arxiv.org/pdf/2212.11116.
[27] W. Ai, Z. Wang, H. Shao, T. Meng, and K. Li, “A Multi-semantic PassingFramework for Semi-supervised Long Text Classification,” Appl. Intell., vol. 53, no. 17, pp. 20174–20190, 2023, doi: 10.1007/s10489-023-04556-x, doi: 10.1007/s10489-023-04556-x.
[28] M. Candito, “Auxiliary Tasks to Boost Biaffine Semantic Dependency Parsing,” Trait. Autom. des Langues Nat. TALN 2022 - Actes la 29e Conf. sur le Trait. Autom. des Langues Nat. Conf. Princ., vol. 1, pp. 424–433, 2022, doi: 10.18653/v1/2022.findings-acl.190.
[29] T. T. Goh, N. A. A. Jamaludin, H. Mohamed, M. N. Ismail, and H. Chua, “Semantic Similarity Analysis for Examination Questions Classification Using WordNet,” Appl. Sci., vol. 13, no. 14, 2023, doi: 10.3390/app13148323.
[30] B. Li, Y. Fan, Y. Sataer, and Z. Gao, “A Higher-Order Semantic Dependency Parser,” ACL, vol. 1, pp. 1–8, 2022, [Online]. Available at: https://arxiv.org/abs/2201.11312.
[31] X. Duan, H. Zan, X. Bai, and C. Zähner, “Reusable Phrase Extraction Based on Syntactic Parsing,” 19th Chinese Natl. Conf. Comput. Linguist. CCL 2020, pp. 1166–1171, 2020, doi: 10.1007/978-3-030-63031-7_33.
[32] A. Basirat and J. Nivre, “Syntactic nuclei in dependency parsing - A multilingual exploration,” EACL 2021 - 16th Conf. Eur. Chapter Assoc. Comput. Linguist. Proc. Conf., no. 2000, pp. 1376–1387, 2021, doi: 10.18653/v1/2021.eacl-main.117.
[33] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of Tricks for Efficient Text Classification,” Proc. ofthe 15th Conf. ofthe Eur. Chapter ofthe Assoc. Comput. Linguist., vol. 2, pp. 427–431, 2017, doi: 10.18653/v1/E17-2068.
[34] O. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient Estimation of Word Representations in Vector Space,” arXiv1301.3781v3 [cs.CL] 7 Sep 2013, pp. 1–12, 2013, [Online]. Available at: https://arxiv.org/abs/1301.3781.
[35] T. Zhou, Y. Wang, and X. Zheng, “Chinese Text Classification Method using FastText and Term Frequency-Inverse Document Frequency Optimization,” J. Phys. Conf. Ser., vol. 1693, no. 1–7, 2020, doi: 10.1088/1742-6596/1693/1/012121.
[36] H. M. Linh, N. T. M. Huyen, V. X. Luong, N. T. Luong, P. T. Hue, and L. Van Cuong, “VLSP 2020 Shared Task: Universal Dependency Parsing for Vietnamese,” Proc. 7th Int. Work. Vietnamese Lang. Speech Process., pp. 77–83, 2020. [Online]. Available at: https://aclanthology.org/2020.vlsp-1.15/.
[37] J. Choi and S. W. Lee, “Improving FastText with Inverse Document Frequency of Subwords,” Pattern Recognit. Lett., vol. 133, pp. 165–172, 2020, doi: 10.1016/j.patrec.2020.03.003.
[38] K. Maity, A. Kumar, and S. Saha, “Attention Based BERT-FastText Model for Hate Speech and Offensive Content Identification in English and Hindi Languages,” CEUR Workshop Proc., vol. 3159, pp. 182–190, 2021. [Online]. Available at: https://ceur-ws.org/Vol-3159/T1-18.pdf.
[39] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, and A. Joulin, “Advances in Pre-training Distributed word Representations,” Lr. 2018 - 11th Int. Conf. Lang. Resour. Eval., no. 1, pp. 52–55, 2017. [Online]. Available at: https://arxiv.org/abs/1712.09405.
[40] A. Vaswani et al., “Attention Is All You Need Ashish,” 31st Conf. Neural Inf. Process. Syst. (NIPS 2017), vol. 8, no. 1, pp. 8–15, 2017, doi: 10.1109/2943.974352.
[41] A. G. D’Sa, I. Illina, and D. Fohr, “BERT and fastText Embeddings for Automatic Detection of Toxic Speech,” Proc. 2020 Int. Multi-Conference Organ. Knowl. Adv. Technol. OCTA 2020, 2020, doi: 10.1109/OCTA49274.2020.9151853.
[42] M. Liang and T. Niu, “Research on Text Classification Techniques Based on Improved TF-IDF Algorithm and LSTM Inputs,” Procedia Comput. Sci., vol. 208, pp. 460–470, 2022, doi: 10.1016/j.procs.2022.10.064.
[43] A. L. I. S. Alammary, “Arabic Questions Classification Using Modified TF-IDF,” IEEE Access, vol. 9, pp. 95109–95122, 2021, doi: 10.1109/ACCESS.2021.3094115.
[44] I. K. Nti, O. Nyarko-Boateng, and J. Aning, “Performance of Machine Learning Algorithms with Different K Values in K-fold CrossValidation,” Int. J. Inf. Technol. Comput. Sci., vol. 13, no. 6, pp. 61–71, Dec. 2021, doi: 10.5815/ijitcs.2021.06.05.
[45] A. Panesar, “Evaluating Machine Learning Models,” in Machine Learning and AI for Healthcare, Berkeley, CA: Apress, 2021, pp. 189–205, doi: 10.1007/978-1-4842-6537-6_7.
[46] S. Raschka, “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning,” arXiv, pp. 1–49, 2018, [Online]. Available at: https://arxiv.org/abs/1811.12808.
[47] J. Wei, C. Huang, S. Vosoughi, and J. Wei, “What are People Asking About COVID-19? A Question Classification Dataset,” Proc. Annu. Meet. Assoc. Comput. Linguist., pp. 1–8, 2020, [Online]. Available at: https://aclanthology.org/2020.nlpcovid19-acl.8/.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
___________________________________________________________
International Journal of Advances in Intelligent Informatics
ISSN 2442-6571 (print) | 2548-3161 (online)
Organized by UAD and ASCEE Computer Society
Published by Universitas Ahmad Dahlan
W: http://ijain.org
E: info@ijain.org (paper handling issues)
andri.pranolo.id@ieee.org (publication issues)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0