(Faculty of Computer Science, Universitas Muslim Indonesia, Indonesia)(2) Adam Adnan
(Faculty of Computer Science, Universitas Muslim Indonesia, Indonesia)(3) Herdianti Darwis
(Faculty of Computer Science, Universitas Muslim Indonesia, Indonesia)(4) Aji Prasetya Wibawa
(Universitas Negeri Malang, Indonesia)(5) Triyanna Widyaningtyas
(Universitas Negeri Malang, Indonesia)(6) Haviluddin Haviluddin
(Universitas Mulawarman, Indonesia)*corresponding author
AbstractFacial Expression Recognition (FER) plays a crucial role in human-computer interaction, yet improving its accuracy remains a significant challenge. This study aims to enhance the robustness and effectiveness of FER systems by integrating multiple machine learning techniques within a semi-supervised learning framework. The primary objective is to develop a more effective ensemble model that combines Long Short-Term Memory (LSTM), Convolutional Neural Networks (CNN), Support Vector Classifier (SVC), and Random Forest classifiers, utilizing both labeled and unlabeled data. The research implements data augmentation and feature extraction techniques, utilizing advanced architectures such as VGG19, ResNet50, and InceptionV3 to improve the quality and representation of facial expression data. Evaluations were conducted across three dataset scenarios: original, feature-extracted, and augmented, using various label-to-unlabeled ratios. The results indicate that the ensemble model achieved a notable accuracy improvement of 87% on the augmented dataset compared to individual classifiers and other ensemble methods, demonstrating superior performance in handling occlusions and diverse data conditions. However, several limitations exist. The study’s reliance on the JAFFE dataset may restrict its generalizability, as it may not cover the full range of facial expressions encountered in real-world scenarios. Additionally, the effect of label-to-unlabeled ratios on the model's performance requires further exploration. Computational efficiency and training time were also not evaluated, which are critical considerations for practical implementation. For future research, it is recommended to employ cross-validation methods for more robust performance evaluation, explore additional data augmentation techniques, optimize ensemble configurations, and address the computational efficiency of the model to better advance FER technologies.
KeywordsDeep Learning; Ensemble Learning; Facial Expression Recognition; Machine Learning;
|
DOIhttps://doi.org/10.26555/ijain.v11i1.1880 |
Article metricsAbstract views : 1071 | PDF views : 252 |
Cite |
Full Text Download
|
References
[1] S. Saurav, R. Saini, and S. Singh, “Facial expression recognition using dynamic local ternary patterns with kernel extreme learning machine classifier,†IEEE Access, vol. 9, pp. 120844–120868, 2021, doi: 10.1109/ACCESS.2021.3108029.
[2] Z. Ullah et al., “Emotion recognition from occluded facial images using deep ensemble model,†Comput. Mater. Contin., vol. 73, no. 3, pp. 4465–4487, 2022, doi: 10.32604/cmc.2022.029101.
[3] R. Zhi, C. Zhou, T. Li, S. Liu, and Y. Jin, “Action unit analysis enhanced facial expression recognition by deep neural network evolution,†Neurocomputing, vol. 425, no. xxxx, pp. 135–148, Feb. 2021, doi: 10.1016/j.neucom.2020.03.036.
[4] H. S. Cha and C. H. Im, “Performance enhancement of facial electromyogram-based facial-expression recognition for social virtual reality applications using linear discriminant analysis adaptation,†Virtual Real., vol. 26, no. 1, pp. 385–398, 2022, doi: 10.1007/s10055-021-00575-6.
[5] R. Singh, S. Saurav, T. Kumar, R. Saini, A. Vohra, and S. Singh, “Facial expression recognition in videos using hybrid CNN & ConvLSTM,†Int. J. Inf. Technol., vol. 15, no. 4, pp. 1819–1830, 2023, doi: 10.1007/s41870-023-01183-0.
[6] A. S. Qazi, M. S. Farooq, F. Rustam, M. G. Villar, C. L. RodrÃguez, and I. Ashraf, “Emotion Detection Using Facial Expression Involving Occlusions and Tilt,†Appl. Sci., vol. 12, no. 22, p. 11797, Nov. 2022, doi: 10.3390/app122211797.
[7] S. M. Gowri, A. Rafeeq, and S. Devipriya, “Detection of real-time facial emotions via deep convolution neural network,†Proc. - 5th Int. Conf. Intell. Comput. Control Syst. ICICCS 2021, no. Iciccs, pp. 1033–1037, 2021, doi: 10.1109/ICICCS51141.2021.9432242.
[8] B. Li and D. Lima, “Facial expression recognition via ResNet-50,†Int. J. Cogn. Comput. Eng., vol. 2, pp. 57–64, Jun. 2021, doi: 10.1016/j.ijcce.2021.02.002.
[9] H. Sikkandar and R. Thiyagarajan, “Deep learning based facial expression recognition using improved cat swarm optimization,†J. Ambient Intell. Humaniz. Comput., vol. 12, no. 2, pp. 3037–3053, 2021, doi: 10.1007/s12652-020-02463-4.
[10] S. Begaj, A. O. Topal, and M. Ali, “Emotion recognition based on facial expressions using convolutional neural network (CNN),†in 2020 International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA), Dec. 2020, pp. 58–63, doi: 10.1109/CoNTESA50436.2020.9302866.
[11] L. Zhao, “A Facial Expression Recognition Method Using Two-Stream Convolutional Networks in Natural Scenes,†J. Inf. Process. Syst., vol. 17, no. 2, pp. 399–410, 2021. [Online]. Available at: https://s3.ap-northeast-2.amazonaws.com/journal-home/journal/jips/fullText/582/15.pdf.
[12] A. R. Khan, “Facial emotion recognition using conventional machine learning and deep learning methods: current achievements, analysis and remaining challenges,†Information, vol. 13, no. 6, p. 268, May 2022, doi: 10.3390/info13060268.
[13] S. Umer, R. K. Rout, C. Pero, and M. Nappi, “Facial expression recognition with trade-offs between data augmentation and deep learning features,†J. Ambient Intell. Humaniz. Comput., vol. 13, no. 2019, pp. 721–735, 2022, doi: 10.1007/s12652-020-02845-8.
[14] A. Fnaiech, H. Sahli, M. Sayadi, and P. Gorce, “Fear facial emotion recognition based on angular deviation,†Electron., vol. 10, no. 3, pp. 1–16, 2021, doi: 10.3390/electronics10030358.
[15] E. G. Dada, D. O. Oyewola, S. B. Joseph, O. Emebo, and O. O. Oluwagbemi, “Facial Emotion Recognition and Classification Using the Convolutional Neural Network-10 (CNN-10),†Appl. Comput. Intell. Soft Comput., vol. 2023, pp. 1–19, Oct. 2023, doi: 10.1155/2023/2457898.
[16] M. K. Chowdary, T. N. Nguyen, and D. J. Hemanth, “Deep learning-based facial emotion recognition for human–computer interaction applications,†Neural Comput. Appl., vol. 35, no. 32, pp. 23311–23328, 2023, doi: 10.1007/s00521-021-06012-8.
[17] T. Han, C. Liu, R. Wu, and D. Jiang, “Deep transfer learning with limited data for machinery fault diagnosis,†Appl. Soft Comput. J., vol. 103, p. 107150, 2021, doi: 10.1016/j.asoc.2021.107150.
[18] T. Nguyen and R. Raich, “Incomplete label multiple instance multiple label learning,†IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 3, pp. 1320–1337, 2022, doi: 10.1109/TPAMI.2020.3017456.
[19] S. Latif, R. Rana, S. Khalifa, R. Jurdak, J. Epps, and B. W. Schuller, “Multi-task semi-supervised adversarial autoencoding for speech emotion recognition,†IEEE Trans. Affect. Comput., vol. 13, no. 2, pp. 992–1004, 2022, doi: 10.1109/TAFFC.2020.2983669.
[20] C. Choi et al., “Semi-supervised target classification in multi-frequency echosounder data,†ICES J. Mar. Sci., vol. 78, no. 7, pp. 2615–2627, 2021, doi: 10.1093/icesjms/fsab140.
[21] K. Zhang, H. Wang, W. Liu, M. Li, J. Lu, and Z. Liu, “An efficient semi-supervised manifold embedding for crowd counting,†Appl. Soft Comput., vol. 96, p. 106634, Nov. 2020, doi: 10.1016/j.asoc.2020.106634.
[22] H. Seyed Alinezhad, J. Shang, and T. Chen, “Early classification of industrial alarm floods based on semisupervised learning,†IEEE Trans. Ind. Informatics, vol. 18, no. 3, pp. 1845–1853, Mar. 2022, doi: 10.1109/TII.2021.3081417.
[23] M. Srinivas, S. Saurav, A. Nayak, and A. P. Murukessan, “Facial expression recognition using fusion of deep learning and multiple features,†Mach. Learn. Algorithms Appl., pp. 229–246, 2021, doi: 10.1002/9781119769262.ch13.
[24] R. Guo, Y. Peng, W. Kong, and F. Li, “A semi-supervised label distribution learning model with label correlations and data manifold exploration,†J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 10, pp. 10094–10108, 2022, doi: 10.1016/j.jksuci.2022.10.008.
[25] E. Lashgari, D. Liang, and U. Maoz, “Data augmentation for deep-learning-based electroencephalography,†J. Neurosci. Methods, vol. 346, no. February, p. 108885, Dec. 2020, doi: 10.1016/j.jneumeth.2020.108885.
[26] Y. Li, X. Yan, Z. Wang, B. Zhang, and Z. Jia, “Clear the fog of negative emotions: A new challenge for intervention towards drug users,†J. Affect. Disord., vol. 294, no. July, pp. 305–313, 2021, doi: 10.1016/j.jad.2021.07.029.
[27] J. Kim and D. Lee, “Facial Expression Recognition Robust to Occlusion and to Intra-Similarity Problem Using Relevant Subsampling,†Sensors, vol. 23, no. 5, p. 2619, Feb. 2023, doi: 10.3390/s23052619.
[28] A. Lumini, L. Nanni, and G. Maguolo, “Deep learning for plankton and coral classification,†Appl. Comput. Informatics, vol. 19, no. 3–4, pp. 265–283, 2023, doi: 10.1016/j.aci.2019.11.004.
[29] H. Saleh, S. Mostafa, A. Alharbi, S. El-Sappagh, and T. Alkhalifah, “Heterogeneous ensemble deep learning model for enhanced arabic sentiment analysis,†Sensors, vol. 22, no. 10, pp. 1–28, 2022, doi: 10.3390/s22103707.
[30] A. Kortylewski, Q. Liu, A. Wang, Y. Sun, and A. Yuille, “Compositional convolutional neural networks: A robust and interpretable model for object recognition under occlusion,†Int. J. Comput. Vis., vol. 129, no. 3, pp. 736–760, Mar. 2021, doi: 10.1007/s11263-020-01401-3.
[31] J. Park, W. Kang, H. Il Koo, and N. I. Cho, “Face Swapping for Low-Resolution and Occluded Images In-the-Wild,†IEEE Access, vol. 12, pp. 91383–91395, 2024, doi: 10.1109/ACCESS.2024.3421528.
[32] M. Bansal, M. Kumar, M. Sachdeva, and A. Mittal, “Transfer learning for image classification using VGG19: Caltech-101 image data set,†J. Ambient Intell. Humaniz. Comput., vol. 14, no. 4, pp. 3609–3620, 2023, doi: 10.1007/s12652-021-03488-z.
[33] S. Sethi, M. Kathuria, and T. Kaushik, “Face mask detection using deep learning : An approach to reduce risk of coronavirus spread,†J. Biomed. Inform., vol. 120, no. September 2020, p. 103848, 2021, doi: 10.1016/j.jbi.2021.103848.
[34] K. Liu, S. Yu, and S. Liu, “An improved inceptionV3 network for obscured ship classification in remote sensing images,†IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 13, pp. 4738–4747, 2020, doi: 10.1109/JSTARS.2020.3017676.
[35] V. R. Joseph, “Optimal ratio for data splitting,†Stat. Anal. Data Min. ASA Data Sci. J., vol. 15, no. 4, pp. 531–538, 2022, doi: 10.1002/sam.11583.
[36] G. Lantzanakis, Z. Mitraka, and N. Chrysoulakis, “X-SVM: An extension of C-SVM algorithm for classification of high-resolution satellite imagery,†IEEE Trans. Geosci. Remote Sens., vol. 59, no. 5, pp. 3805–3815, May 2021, doi: 10.1109/TGRS.2020.3017937.
[37] H. Darwis, Z. Ali, Y. Salim, and P. L. L. Belluano, “Max feature map cnn with support vector guided softmax for face recognition,†Int. J. Informatics Vis., vol. 7, no. 3, pp. 959–966, 2023, doi: 10.30630/joiv.7.3.1751.
[38] G. Van Houdt, C. Mosquera, and G. Nápoles, “A review on the long short-term memory model,†Artif. Intell. Rev., vol. 53, no. 8, pp. 5929–5955, Dec. 2020, doi: 10.1007/s10462-020-09838-1.
[39] H. Yu, L. T. Yang, Q. Zhang, D. Armstrong, and M. J. Deen, “Neurocomputing convolutional neural networks for medical image analysis : State-of-the- art , comparisons , improvement and perspectives,†Neurocomputing, vol. 444, pp. 92–110, 2021, doi: 10.1016/j.neucom.2020.04.157.
[40] M. Mafarja, T. Thaher, M. A. Al-betar, I. A. Doush, and H. Turabieh, “Classification framework for faulty-software using enhanced exploratory whale optimizer-based feature selection scheme and random forest ensemble learning,†pp. 18715–18757, 2023, doi: 10.1007/s10489-022-04427-x.
[41] W. Wattanapornprom, C. Thammarongtham, A. Hongsthong, and S. Lertampaiporn, “Ensemble of multiple classifiers for multilabel classification of plant protein subcellular localization,†Life, vol. 11, no. 4, p. 293, Mar. 2021, doi: 10.3390/life11040293.
[42] L. Zou, H. Wu, R. Liu, C. Yi, J. He, and Y. Li, “A new method for LDPC blind recognition over a candidate set using Kullback-Leibler divergence,†IEEE Commun. Lett., vol. 28, no. 5, pp. 964–968, May 2024, doi: 10.1109/LCOMM.2024.3373074.
[43] P.-J. Chao et al., “Using deep learning models to analyze the cerebral edema complication caused by radiotherapy in patients with intracranial tumor,†Sci. Rep., vol. 12, no. 1, p. 1555, 2022, doi: 10.1038/s41598-022-05455-w.
[44] W. Hu, L. Wu, M. Jian, Y. Chen, and H. Yu, “Cosine metric supervised deep hashing with balanced similarity,†Neurocomputing, vol. 448, pp. 94–105, 2021, doi: 10.1016/j.neucom.2021.03.093.
[45] Z. E. Huma et al., “A hybrid deep random neural network for cyberattack detection in the industrial internet of ihings,†IEEE Access, vol. 9, pp. 55595–55605, 2021, doi: 10.1109/ACCESS.2021.3071766.
[46] W. M. Alenazy and A. S. Alqahtani, “Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition,†J. Ambient Intell. Humaniz. Comput., vol. 12, no. 2, pp. 1631–1646, 2021, doi: 10.1007/s12652-020-02235-0.
[47] J. Mao, Y. Hu, and R. Wan, “Face Expression Recognition Method Based on Weak Annotation and Conditional Generative Adversarial Neural Networks,†The Lancent Pschch. pp. 1–35, 2024, doi: 10.2139/ssrn.4820797.
[48] A. I. Jabbooree, L. M. Khanli, P. Salehpour, and S. Pourbahrami, “Geometrical Facial Expression Recognition Approach Based on Fusion CNN-SVM,†Int. J. Intell. Eng. Syst., vol. 17, no. 1, pp. 457–468, 2024, doi: 10.22266/ijies2024.0229.40.
[49] M. Hu, C. Yang, Y. Zheng, X. Wang, L. He, and F. Ren, “Facial Expression Recognition Based on Fusion Features of Center-Symmetric Local Signal Magnitude Pattern,†IEEE Access, vol. 7, pp. 118435–118445, 2019, doi: 10.1109/ACCESS.2019.2936976.
[50] S. Yuan and X. Mao, “Exponential elastic preserving projections for facial expression recognition,†Neurocomputing, vol. 275, pp. 711–724, 2018, doi: 10.1016/j.neucom.2017.08.067.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
___________________________________________________________
International Journal of Advances in Intelligent Informatics
ISSN 2442-6571 (print) | 2548-3161 (online)
Organized by UAD and ASCEE Computer Society
Published by Universitas Ahmad Dahlan
W: http://ijain.org
E: info@ijain.org (paper handling issues)
andri.pranolo.id@ieee.org (publication issues)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0