(Universitas Muhammadiyah Makassar, Indonesia)(2) Adriani Adriani
(Universitas Muhammadiyah Makassar, Indonesia)(3) Rahmania rahmania
(Universitas Muhammadiyah Makassar, Indonesia)(4) Mus’ab Sahrim
(Universiti Sains Islam Malaysia, Malaysia)(5) Asep Indra Syahyadi
(Universitas Islam Negeri Alauddin Makassar, Indonesia)(6) Haris Setiaji
(Institut Agama Islam Negeri Metro Lampung, Indonesia)*corresponding author
AbstractSign Language Recognition (SLR) helps deaf people communicate with normal people. However, SLR still has difficulty detecting dynamic movements of connected sign language, which reduces the accuracy of detection. This results from a sentence's usage of transitional gestures between words. Several researchers have tried to solve the problem of transition gestures in dynamic sign language, but none have been able to produce an accurate solution. The R-GB LSTM method detects transition gestures within a sentence based on labelled words and transition gestures stored in a model. If a gesture to be processed during training matches a transition gesture stored in the pre-training process and its probability value is greater than 0.5, it is categorized as a transition gesture. Subsequently, the detected gestures are eliminated according to the gesture's time value (t). To evaluate the effectiveness of the proposed method, we conducted an experiment using 20 words in Indonesian Sign Language (SIBI). Twenty representative words were selected for modelling using our R-GB LSTM technique. The results are promising, with an average accuracy of 80% for gesture sentences and an even more impressive accuracy rate of 88.57% for gesture words. We used a confusion matrix to calculate accuracy, specificity, and sensitivity. This study marks a significant leap forward in developing sustainable sign language recognition systems with improved accuracy and practicality. This advancement holds great promise for enhancing communication and accessibility for deaf and hard-of-hearing communities.
KeywordsDeaf People; R-GB LSTM; Word Sign; Sentence Sign; Sign Language
|
DOIhttps://doi.org/10.26555/ijain.v10i2.1445 |
Article metricsAbstract views : 239 | PDF views : 53 |
Cite |
Full Text Download
|
References
[1] C. Hinchcliffe et al., “Language comprehension in the social brain: Electrophysiological brain signals of social presence effects during syntactic and semantic sentence processing,” Cortex, vol. 130, pp. 413–425, Sep. 2020, doi: 10.1016/j.cortex.2020.03.029.
[2] A. A. Zare and S. H. Zahiri, “Recognition of a real-time signer-independent static Farsi sign language based on fourier coefficients amplitude,” Int. J. Mach. Learn. Cybern., vol. 9, no. 5, pp. 727–741, May 2018, doi: 10.1007/s13042-016-0602-3.
[3] M. Al-Qurishi, T. Khalid, and R. Souissi, “Deep Learning for Sign Language Recognition: Current Techniques, Benchmarks, and Open Issues,” IEEE Access, vol. 9, pp. 126917–126951, 2021, doi: 10.1109/ACCESS.2021.3110912.
[4] A. Chaikaew, K. Somkuan, and T. Yuyen, “Thai Sign Language Recognition: an Application of Deep Neural Network,” in 2021 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunication Engineering, Mar. 2021, pp. 128–131, doi: 10.1109/ECTIDAMTNCON51128.2021.9425711.
[5] M. Alaghband, H. R. Maghroor, and I. Garibay, “A survey on sign language literature,” Mach. Learn. with Appl., vol. 14, p. 100504, Dec. 2023, doi: 10.1016/j.mlwa.2023.100504.
[6] A. Abbaskhah, H. Sedighi, and H. Marvi, “Infant cry classification by MFCC feature extraction with MLP and CNN structures,” Biomed. Signal Process. Control, vol. 86, p. 105261, Sep. 2023, doi: 10.1016/j.bspc.2023.105261.
[7] P. K. Athira, C. J. Sruthi, and A. Lijiya, “A Signer Independent Sign Language Recognition with Co-articulation Elimination from Live Videos: An Indian Scenario,” J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 3, pp. 771–781, Mar. 2022, doi: 10.1016/j.jksuci.2019.05.002.
[8] Z. Liang, H. Li, and J. Chai, “Sign Language Translation: A Survey of Approaches and Techniques,” Electronics, vol. 12, no. 12, p. 2678, Jun. 2023, doi: 10.3390/electronics12122678.
[9] S. Kausar and M. Y. Javed, “A Survey on Sign Language Recognition,” in 2011 Frontiers of Information Technology, Dec. 2011, pp. 95–98, doi: 10.1109/FIT.2011.25.
[10] I. A. Adeyanju, O. O. Bello, and M. A. Adegboye, “Machine learning methods for sign language recognition: A critical review and analysis,” Intell. Syst. with Appl., vol. 12, p. 200056, Nov. 2021, doi: 10.1016/j.iswa.2021.200056.
[11] A. Choudhury, A. Kumar Talukdar, M. Kamal Bhuyan, and K. Kumar Sarma, “Movement Epenthesis Detection for Continuous Sign Language Recognition,” J. Intell. Syst., vol. 26, no. 3, pp. 471–481, Jul. 2017, doi: 10.1515/jisys-2016-0009.
[12] B. Mocialov, G. Turner, K. Lohan, and H. Hastie, “Towards Continuous Sign Language Recognition with Deep Learning,” Proceeding Work. Creat. Mean. with Robot Assist. Gap Left by Smart Device, p. 5, 2017, [Online]. Available at: https://homepages.inf.ed.ac.uk/hhastie2/pubs/humanoids.pdf.
[13] J. P. Sahoo, S. Ari, and S. K. Patra, “A user independent hand gesture recognition system using deep CNN feature fusion and machine learning technique,” in New Paradigms in Computational Modeling and Its Applications, Elsevier, 2021, pp. 189–207, doi: 10.1016/B978-0-12-822133-4.00011-6.
[14] A. S. Agrawal, A. Chakraborty, and C. M. Rajalakshmi, “Real-Time Hand Gesture Recognition System Using MediaPipe and LSTM,” Int. J. Res. Publ. Rev., vol. 3, no. 4, pp. 2509–2515, 2022, [Online]. Available at: https://ijrpr.com/uploads/V3ISSUE4/IJRPR3693.pdf.
[15] J. Bora, S. Dehingia, A. Boruah, A. A. Chetia, and D. Gogoi, “Real-time Assamese Sign Language Recognition using MediaPipe and Deep Learning,” Procedia Comput. Sci., vol. 218, pp. 1384–1393, Jan. 2023, doi: 10.1016/j.procs.2023.01.117.
[16] E. R. Swedia, A. B. Mutiara, M. Subali, and Ernastuti, “Deep Learning Long-Short Term Memory (LSTM) for Indonesian Speech Digit Recognition using LPC and MFCC Feature,” in 2018 Third International Conference on Informatics and Computing (ICIC), Oct. 2018, pp. 1–5, doi: 10.1109/IAC.2018.8780566.
[17] T. S. Dias, J. J. A. Mendes, and S. F. Pichorim, “Comparison between handcraft feature extraction and methods based on Recurrent Neural Network models for gesture recognition by instrumented gloves: A case for Brazilian Sign Language Alphabet,” Biomed. Signal Process. Control, vol. 80, p. 104201, Feb. 2023, doi: 10.1016/j.bspc.2022.104201.
[18] K. Anand, S. Urolagin, and R. K. Mishra, “How does hand gestures in videos impact social media engagement - Insights based on deep learning,” Int. J. Inf. Manag. Data Insights, vol. 1, no. 2, p. 100036, Nov. 2021, doi: 10.1016/j.jjimei.2021.100036.
[19] A. A. Ilham, I. Nurtanio, Ridwang, and Syafaruddin, “Applying LSTM and GRU Methods to Recognize and Interpret Hand Gestures, Poses, and Face-Based Sign Language in Real Time,” J. Adv. Comput. Intell. Intell. Informatics, vol. 28, no. 2, pp. 265–272, Mar. 2024, doi: 10.20965/jaciii.2024.p0265.
[20] Ridwang, I. Nurtanio, A. A. Ilham, and Syafaruddin, “Deaf Sign Language Translation System With Pose and Hand Gesture Detection Under Lstm-Sequence Classification Model,” ICIC Express Lett., vol. 17, no. 7, pp. 809–816, 2023. [Online]. Available at: https://cir.nii.ac.jp/crid/1390296265971342976.
[21] S. C. Agrawal, A. S. Jalal, and C. Bhatnagar, “Recognition of Indian Sign Language using feature fusion,” in 2012 4th International Conference on Intelligent Human Computer Interaction (IHCI), Dec. 2012, pp. 1–5, doi: 10.1109/IHCI.2012.6481841.
[22] Q. Xiao, X. Chang, X. Zhang, and X. Liu, “Multi-Information Spatial–Temporal LSTM Fusion Continuous Sign Language Neural Machine Translation,” IEEE Access, vol. 8, pp. 216718–216728, 2020, doi: 10.1109/ACCESS.2020.3039539.
[23] Z. Zhang, C. Xu, J. Xie, Y. Zhang, P. Liu, and Z. Liu, “MFCC-LSTM framework for leak detection and leak size identification in gas-liquid two-phase flow pipelines based on acoustic emission,” Measurement, vol. 219, p. 113238, Sep. 2023, doi: 10.1016/j.measurement.2023.113238.
[24] Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grundmann, “Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs,” pp. 1-5, 2019. [Online]. Available at: https://static1.squarespace.com/static/5c3f69e1cc8fedbc039ea739/t/5d015ff133d4280001167610/1560371186690/6_CV4AR_Mesh.pdf.
[25] A. G. Salman, Y. Heryadi, E. Abdurahman, and W. Suparta, “Single Layer & Multi-layer Long Short-Term Memory (LSTM) Model with Intermediate Variables for Weather Forecasting,” Procedia Comput. Sci., vol. 135, pp. 89–98, Jan. 2018, doi: 10.1016/j.procs.2018.08.153.
[26] B. Sundar and T. Bagyammal, “American Sign Language Recognition for Alphabets Using MediaPipe and LSTM,” Procedia Comput. Sci., vol. 215, pp. 642–651, Jan. 2022, doi: 10.1016/j.procs.2022.12.066.
[27] A. Mittal, P. Kumar, P. P. Roy, R. Balasubramanian, and B. B. Chaudhuri, “A Modified LSTM Model for Continuous Sign Language Recognition Using Leap Motion,” IEEE Sens. J., vol. 19, no. 16, pp. 7056–7063, Aug. 2019, doi: 10.1109/JSEN.2019.2909837.
[28] M. A. As’ari, N. A. J. Sufri, and G. S. Qi, “Emergency sign language recognition from variant of convolutional neural network (CNN) and long short term memory (LSTM) models,” Int. J. Adv. Intell. Informatics, vol. 10, no. 1, p. 64, Feb. 2024, doi: 10.26555/ijain.v10i1.1170.
[29] Ridwang, A. A. Ilham, I. Nurtanio, and - Syafaruddin, “Dynamic Sign Language Recognition Using Mediapipe Library and Modified LSTM Method,” Int. J. Adv. Sci. Eng. Inf. Technol., vol. 13, no. 6, pp. 2171–2180, Dec. 2023, doi: 10.18517/ijaseit.13.6.19401.
[30] C. Millar, N. Siddique, and E. Kerr, “LSTM Network Classification of Dexterous Individual Finger Movements,” J. Adv. Comput. Intell. Intell. Informatics, vol. 26, no. 2, pp. 113–124, Mar. 2022, doi: 10.20965/jaciii.2022.p0113.
[31] W. Abdul et al., “Intelligent real-time Arabic sign language classification using attention-based inception and BiLSTM,” Comput. Electr. Eng., vol. 95, p. 107395, Oct. 2021, doi: 10.1016/j.compeleceng.2021.107395.
[32] J. Fayyad, M. A. Jaradat, D. Gruyer, and H. Najjaran, “Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review,” Sensors, vol. 20, no. 15, p. 4220, Jul. 2020, doi: 10.3390/s20154220.
[33] E. Pan, X. Mei, Q. Wang, Y. Ma, and J. Ma, “Spectral-spatial classification for hyperspectral image based on a single GRU,” Neurocomputing, vol. 387, pp. 150–160, Apr. 2020, doi: 10.1016/j.neucom.2020.01.029.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
___________________________________________________________
International Journal of Advances in Intelligent Informatics
ISSN 2442-6571 (print) | 2548-3161 (online)
Organized by UAD and ASCEE Computer Society
Published by Universitas Ahmad Dahlan
W: http://ijain.org
E: info@ijain.org (paper handling issues)
andri.pranolo.id@ieee.org (publication issues)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0