pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning

Authors

DOI:

https://doi.org/10.52825/ocp.v11i.3663

Keywords:

Ontology Learning, Large Language Models, Retrieval-Augmented Generation (RAG), Vocabulary-Constrained Filtering, Knowledge Representation

Abstract

Ontology learning from text remains challenging despite significant progress in Large Language Models (LLMs), which can hallucinate domain terms, produce inconsistent formats, and favor hierarchical over associative relations. In the LLMs4OL 2026 Challenge, we address both the End-to-End Flagship Task (Task A) and Ontology Extension Reuse Task (Task B) using an offline retrieval-augmented few-shot prompting pipeline. Our system employs Qwen2.5-14B-Instruct with all-MiniLM-L6-v2 for demonstration retrieval, selecting the top-5 examples for Task A and top-2 for Task B. A left-truncated context-windowing strategy preserves task instructions within long prompts. For Task B, generated triples undergo deterministic vocabulary-constrained filtering, retaining triples when at least one endpoint belongs to the sample’s closed term/type vocabulary and removing duplicates of the initial ontology. The approach achieves Semantic Graph Similarity of 0.8692, Term-Typing F1 of 0.9200, and Taxonomy Discovery F1 of 0.8540 on Task B, while Task A achieves 0.7416 Semantic Graph Similarity. However, no non-taxonomic relations are extracted, highlighting limitations of closed, taxonomy-oriented relation vocabularies.

Downloads

Download data is not yet available.

References

[1] J. Li, D. Garijo, and M. Poveda-Villalón, “Large language models for ontology engineering: A systematic literature review,” Semantic Web, vol. 17, no. 4, p. 22 104 968 261 465 514, 2026.

[2] T. Aggarwal, A. Salatino, F. Osborne, and E. Motta, “Large language models for scholarly ontology generation: An extensive analysis in the engineering field,” Information Processing & Management, vol. 63, no. 1, p. 104 262, 2026.

[3] O. Perera and J. Liu, “Exploring large language models for ontology learning,” 2024.

[4] H. Terdalkar and A. Bhattacharya, “Framework for question-answering in sanskrit through automated construction of knowledge graphs,” in Proceedings of the 6th International Sanskrit Computational Linguistics Symposium, 2019, pp. 97–116.

[5] H. Babaei Giglou, J. D’Souza, and S. Auer, “Llms4ol: Large language models for ontology learning,” in The Semantic Web – ISWC 2023, T. R. Payne et al., Eds., Cham: Springer Nature Switzerland, 2023, pp. 408–427, ISBN: 978-3-031-47240-4.

[6] H. B. Giglou, J. D’Souza, and S. Auer, “Llms4ol 2024 overview: The 1st large language models for ontology learning challenge,” Open Conference Proceedings, vol. 4, pp. 3–16, Oct. 2024. DOI: 10.52825/ocp.v4i.2473 [Online]. Available: https://www.tib-op.org/ojs/index.php/ocp/article/view/2473

[7] H. B. Giglou, J. D’Souza, N. Mihindukulasooriya, and S. Auer, “Llms4ol 2025 overview: The 2nd large language models for ontology learning challenge,” Open Conference Proceedings, vol. 6, Oct. 2025. DOI: 10.52825/ocp.v6i.2913 [Online]. Available: https://www.tib-op.org/ojs/index.php/ocp/article/view/2913

[8] H. B. Giglou, J. D’Souza, and S. Auer, “Llms4ol 2026 overview: The 3rd large language models for ontology learning challenge,” Open Conference Proceedings, 2026.

[9] R. M. Bakker, D. L. Di Scala, M. H. de Boer, and S. A. Raaijmakers, “Ontology learning with llms: A benchmark study on axiom identification,” arXiv preprint arXiv:2512.05594, 2025.

[10] C. Yimmark and T. Racharak, “T-grec at llms4ol 2025 task b: A report on term-typing task of obi dataset using llm with k-nearest neighbors,” in Open Conference Proceedings, vol. 6, 2025.

[11] A. Chourasia, A. Ghosh, and M. Ojha, “Balancing lexical and semantic models for optimized biomedical retrieval,” in 2025 IEEE 12th Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON), 2025, pp. 1–6. DOI: 10.1109/UPCON66414.2025.11453897

[12] H. B. Giglou, J. D’Souza, S. Sadruddin, and S. Auer, “Llms4ol 2024 datasets: Toward ontology learning with large language models,” in Open Conference Proceedings, vol. 4, 2024, pp. 17–30.

[13] A. Beliaeva and T. Rahmatullaev, “Heterogeneous llm methods for ontology learning (few-shot prompting, ensemble typing, and attention-based taxonomies),” arXiv preprint arXiv:2508.19428, 2025.

[14] P. K. Goyal, S. Singh, and U. S. Tiwary, “Silp nlp at llms4ol 2025 tasks a, b, c, and d: Clustering-based ontology learning using llms,” in Open Conference Proceedings, vol. 6, 2025.

Downloads

Published

2026-11-04

How to Cite

Mishra, S., Ram Meena, D., Ojha, M., Pratap Singh, K., & Singh, K. (2026). pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning. Open Conference Proceedings, 11. https://doi.org/10.52825/ocp.v11i.3663

Conference Proceedings Volume

Section

LLMs4OL 2026 Task Participant Long Papers