(Faculty of Computer Science, Universitas Indonesia, Indonesia)(2) Fariz Darari
(Faculty of Computer Science, Universitas Indonesia, Indonesia)(3) Reynard Adha Ryanda
(Faculty of Computer Science, Universitas Indonesia, Indonesia)*corresponding author
AbstractQuestion answering (QA) technologies are crucial for building conversational AI. Current research related to QA for the legal domain lacks focus on the organized structure of laws, which are hierarchically segmented into components at varying levels of detail. To address this gap, we propose a new task of granularity-aware legal QA, which accounts for the underlying granularity levels of law components. Our approach encompasses task formulation, dataset creation, and model development. Under the Indonesian jurisdiction, we consider four law component granularity levels: chapters (bab), articles (pasal), sections (ayat), and letters (huruf). We include 15 government regulations (Peraturan Pemerintah) of Indonesia related to labor affairs and build a legal QA dataset with granularity information. We then design a solution for such a task—the first IR system to account for legal component granularity. We implement a customized retriever-reranker pipeline in which the retriever accepts law components of multiple granularities and the reranker is trained for granularity-aware ranking. We leverage BM25 and BERT models as retriever and reranker, respectively, yielding an end-to-end exact match accuracy of 35.68%, which offers a significant improvement (20%) over a strong baseline. The use of reranker also improves the granularity accuracy from 44.86% to 63.24%. In practical context, such a solution can help provide more precise answers, not only from legal chatbots, but also other conversational AI that deals with hierarchically-structured documents.
KeywordsGranularity-aware; Question answering; Retrieval; Regulation; BERT
|
DOIhttps://doi.org/10.26555/ijain.v10i3.1105 |
Article metricsAbstract views : 490 | PDF views : 150 |
Cite |
Full Text Download
|
References
[1] W. Lehnert, “Human and Computational Question Answering*,” Cogn. Sci., vol. 1, no. 1, pp. 47–73, Jan. 1977, doi: 10.1207/s15516709cog0101_3.
[2] P. Rajpurkar, R. Jia, and P. Liang, “Know What You Don’t Know: Unanswerable Questions for SQuAD,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018, vol. 2, pp. 784–789, doi: 10.18653/v1/P18-2124.
[3] R. Karra and A. Lasfar, “Analysis of QA System Behavior against Context and Question Changes,” Int. Arab J. Inf. Technol., vol. 21, no. 2, pp. 191–200, 2024, doi: 10.34028/iajit/21/2/2.
[4] Z. Yang et al., “HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2369–2380, doi: 10.18653/v1/D18-1259.
[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North, 2019, pp. 4171–4186, doi: 10.18653/v1/N19-1423.
[6] V. Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 6769–6781, doi: 10.18653/v1/2020.emnlp-main.550.
[7] J. Moreno Schneider et al., “Lynx: A knowledge-based AI service platform for content processing, enrichment and analysis for the legal domain,” Inf. Syst., vol. 106, p. 101966, May 2022, doi: 10.1016/j.is.2021.101966.
[8] M. Kaltenboeck, P. Boil, P. Verhoeven, C. Sageder, E. Montiel-Ponsoda, and P. Calleja-Ibáñez, “Using a Legal Knowledge Graph for Multilingual Compliance Services in Labor Law, Contract Management, and Geothermal Energy,” in Technologies and Applications for Big Data Value, Cham: Springer International Publishing, 2022, pp. 253–271, doi: 10.1007/978-3-030-78307-5_12.
[9] V. Socatiyanurak et al., “LAW-U: Legal Guidance Through Artificial Intelligence Chatbot for Sexual Violence Victims and Survivors,” IEEE Access, vol. 9, pp. 131440–131461, 2021, doi: 10.1109/ACCESS.2021.3113172.
[10] D. R. Faisal, F. Darari, B. C. L. Tobing, and O. Lee, “Towards Building a Legal Virtual Assistant Based on Knowledge Graphs,” CEUR Workshop Proc., vol. 3257, pp. 73–78, 2022. [Online]. Available at: https://scholar.ui.ac.id/en/publications/towards-building-a-legal-virtual-assistant-based-on-knowledge-gra.
[11] M. Wyawahare, S. Roy, and S. Zanwar, “Generative vs Intent-based Chatbot for Judicial Advice,” in 2024 IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI), Mar. 2024, pp. 1–6, doi: 10.1109/IATMSI60426.2024.10502550.
[12] R. DALE, “Law and Word Order: NLP in Legal Tech,” Nat. Lang. Eng., vol. 25, no. 1, pp. 211–217, Jan. 2019, doi: 10.1017/S1351324918000475.
[13] J. Martinez-Gil, “A survey on legal question–answering systems,” Comput. Sci. Rev., vol. 48, p. 100552, May 2023, doi: 10.1016/j.cosrev.2023.100552.
[14] D. Jurafsky and J. H. Martin, Speech and Language Processing. pp. 1-559, 2024. [Online]. Available at: https://web.stanford.edu/~jurafsky/slp3/.
[15] C. D. Manning, P. Raghavan, and H. Schütze, “Introduction to Information Retrieval,” Introd. to Inf. Retr., Jul. pp. 1-461, 2008, doi: 10.1017/CBO9780511809071.
[16] S. Levy, K. Mo, W. Xiong, and W. Y. Wang, “Open-Domain Question-Answering for COVID-19 and Other Emergent Domains,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2021, pp. 259–266, doi: 10.18653/v1/2021.emnlp-demo.30.
[17] S. P. Widodo, “Comparative Analysis of Retriever and Reader for Open Domain Questions Answering on BPS Knowledge in Indonesian,” Proc. Int. Conf. Data Sci. Off. Stat., vol. 2023, no. 1, pp. 337–343, Dec. 2023, doi: 10.34123/icdsos.v2023i1.384.
[18] N. Abduljaleel, A. Corrada-Emmanuel, Q. Li, X. Liu, C. Wade, and J. Allan, “UMass at TREC 2003: HARD and QA,” TREC, pp. 1–11, 2003, doi: 10.6028/NIST.SP.500-255.qa-umass.allan.
[19] J. Allan, B. Croft, A. Moffat, and M. Sanderson, “Frontiers, challenges, and opportunities for information retrieval,” ACM SIGIR Forum, vol. 46, no. 1, pp. 2–32, May 2012, doi: 10.1145/2215676.2215678.
[20] F. Bu, X. Zhu, Y. Hao, and X. Zhu, “Function-Based Question Classification for General QA,” no. 11. Association for Computational Linguistics, pp. 1119–1128, 2010. [Online]. Available at: https://aclanthology.org/D10-1109.
[21] V. Bolotova, V. Blinov, F. Scholer, W. B. Croft, and M. Sanderson, “A Non-Factoid Question-Answering Taxonomy,” in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Jul. 2022, pp. 1196–1207, doi: 10.1145/3477495.3531926.
[22] A. Peñas et al., “Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 6241 LNCS, Springer, Berlin, Heidelberg, 2010, pp. 174–196, doi: 10.1007/978-3-642-15754-7_21.
[23] J. Rabelo, R. Goebel, M.-Y. Kim, Y. Kano, M. Yoshioka, and K. Satoh, “Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021,” Rev. Socionetwork Strateg., vol. 16, no. 1, pp. 111–133, Apr. 2022, doi: 10.1007/s12626-022-00105-z.
[24] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3980–3990, doi: 10.18653/v1/D19-1410.
[25] I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos, “LEGAL-BERT: The Muppets straight out of Law School,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 2898–2904, doi: 10.18653/v1/2020.findings-emnlp.261.
[26] S. Wehnert, V. Sudhi, S. Dureja, L. Kutty, S. Shahania, and E. W. De Luca, “Legal norm retrieval with variations of the bert model combined with TF-IDF vectorization,” in Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, Jun. 2021, pp. 285–294, doi: 10.1145/3462757.3466104.
[27] B. Mansouri and R. Campos, FALQU: Finding Answers to Legal Questions, vol. 1, no. 1, pp. 1-4. Association for Computing Machinery, 2023. [Online]. Available at: https://arxiv.org/pdf/2304.05611.
[28] A. Askari, Z. Yang, Z. Ren, and S. Verberne, “Answer Retrieval in Legal Community Question Answering,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14610 LNCS, Springer, Cham, 2024, pp. 477–485, doi: 10.1007/978-3-031-56063-7_40.
[29] A. Hogan et al., Knowledge Graphs. Cham: Springer International Publishing, pp. 1-237, 2022. [Online]. Available at: https://link.springer.com/10.1007/978-3-031-01918-0.
[30] S. Gao et al., “How Legal Knowledge Graph Can Help Predict Charges for Legal Text,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14452 LNCS, Springer, Singapore, 2024, pp. 408–420, doi: 10.1007/978-981-99-8076-5_30.
[31] A. Revenko and P. Martín-Chozas, “Extraction and Semantic Representation of Domain-Specific Relations in Spanish Labour Law,” Proces. del Leng. Nat., vol. 69, pp. 105–116, 2022. [Online]. Available at: https://rua.ua.es/dspace/bitstream/10045/127407/1/PLN_69_09.pdf.
[32] J. S. Dhani, R. Bhatt, B. Ganesan, P. Sirohi, and V. Bhatnagar, “Similar Cases Recommendation using Legal Knowledge Graphs,” in SAIL’23: 3rd Symposium on Artificial Intelligence and Law, Jul. 2021, pp. 1–12. [Online]. Available at: https://arxiv.org/abs/2107.04771v2.
[33] M. Abdurahman, F. Darari, H. Lesmana, M. Hartopo, I. Rhesa, and B. C. Lumban Tobing, “Lex2KG: Automatic Conversion of Legal Documents to Knowledge Graph,” in 2021 International Conference on Advanced Computer Science and Information Systems (ICACSIS), Oct. 2021, pp. 1–5, doi: 10.1109/ICACSIS53237.2021.9631310.
[34] A. Hamid and Hasbullah, “Legal Hermeneutics of the Omnibus Law on Jobs Creation: A Case Study in Indonesia,” Beijing Law Rev., vol. 13, no. 03, pp. 449–476, Jul. 2022, doi: 10.4236/blr.2022.133028.
[35] G. Klyne, Jeremy J. Carroll, and B. McBride, RDF 1.1 Concepts and Abstract Syntax. pp. 1-22, 2014. [Online]. Available at: https://www.w3.org/TR/rdf11-concepts/.
[36] S. E. Robertson, S. Walker, S. Jones, and M. Hancock-Beaulieu, “Okapi at TREC-3.,” City, no. January 1994, pp. 1–14, 1994, [Online]. Available at: https://www.researchgate.net/publication/221037764_Okapi_at_TREC-3.
[37] J. Bromley et al., “Signature Verification using a ‘Siamese’ Time Delay Neural Network,” in Advances in Neural Information Processing Systems, 1993, pp. 737–744. [Online]. Available: https://papers.neurips.cc/paper_files/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf.
[38] A. W. Pradana and M. Hayaty, “The Effect of Stemming and Removal of Stopwords on the Accuracy of Sentiment Analysis on Indonesian-language Texts,” Kinet. Game Technol. Inf. Syst. Comput. Network, Comput. Electron. Control, vol. 4, no. 3, pp. 375–380, Oct. 2019, doi: 10.22219/kinetik.v4i4.912.
[39] A. Liu, S. Swayamdipta, N. A. Smith, and Y. Choi, “WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation,” in Findings of the Association for Computational Linguistics: EMNLP 2022, Jan. 2022, pp. 6826–6847, doi: 10.18653/v1/2022.findings-emnlp.508.
[40] F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics, 2020, pp. 757–770, doi: 10.18653/v1/2020.coling-main.66.
[41] B. Wilie et al., “IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding,” in Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, 2020, pp. 843–857. [Online]. Available at: https://aclanthology.org/2020.aacl-main.85.
[42] D. D. Prasetya, A. Prasetya Wibawa, and T. Hirashima, “The performance of text similarity algorithms,” Int. J. Adv. Intell. Informatics, vol. 4, no. 1, p. 63, Mar. 2018, doi: 10.26555/ijain.v4i1.152.
[43] P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Advances in Neural Information Processing Systems, 2020, pp. 1–16. [Online]. Available: https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
___________________________________________________________
International Journal of Advances in Intelligent Informatics
ISSN 2442-6571 (print) | 2548-3161 (online)
Organized by UAD and ASCEE Computer Society
Published by Universitas Ahmad Dahlan
W: http://ijain.org
E: info@ijain.org (paper handling issues)
andri.pranolo.id@ieee.org (publication issues)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0