LLMs4OL 2026 Overview: The 3rd Large Language Models for Ontology Learning Challenge

Authors

DOI:

https://doi.org/10.52825/ocp.v11i.3728

Keywords:

Ontology Learning, OntoLearner, LLMs4OL, Generative AI, Large Language Models

Abstract

We present the results of the 3rd Large Language Models for Ontology Learning Challenge, held at the International Semantic Web Conference 2026, a shared task designed to evaluate the effectiveness of large language models (LLMs) for ontology learning. The challenge comprised three complementary tasks: end-to-end ontology construction from text (Flagship), extension of an existing partial ontology (Reuse), and cross-domain taxonomy discovery (Taxonomy). Participants explored a broad range of approaches, including general-purpose and adapted LLMs, embedding-based retrieval, conventional NLP pipelines, and hybrid symbolic methods. Results show that constraining the label space through a supplied vocabulary matters more than model scale, and that retrieval grounding and task-specific adaptation of open-weight models transferred more reliably than post-hoc correction. Persistent challenges include candidate retrieval, entity normalization, and complete taxonomy recovery, while non-taxonomic relation extraction was rarely attempted under the current scoring. Building on this edition, we are integrating submitted systems into OntoLearner, our modular framework for ontology learning, so challenge contributions remain reusable and comparable.

Downloads

Download data is not yet available.

References

[1] H. Babaei Giglou, J. D’Souza, and S. Auer, “Llms4ol 2024 overview: The 1st large language models for ontology learning challenge,” Open Conference Proceedings, vol. 4, pp. 3–16, Oct. 2024. DOI: 10.52825/ocp.v4i.2473. [Online]. Available: https://www.tib-op.org/ojs/index.php/ocp/article/view/2473.

[2] H. Babaei Giglou, J. D’Souza, N. Mihindukulasooriya, and S. Auer, “LLMs4OL 2025 Overview: The 2nd Large Language Models for Ontology Learning Challenge,” Open Conference Proceedings, vol. 6, Oct. 2025. DOI: 10.52825/ocp.v6i.2913. [Online]. Available: https://www.tib-op.org/ojs/index.php/ocp/article/view/2913.

[3] H. Babaei Giglou, J. D’Souza, and S. Auer, “Llms4ol: Large language models for ontology learning,” in International semantic web conference, Springer, 2023, pp. 408–427.

[4] H. B. Giglou, J. D’Souza, A. Aioanei, N. Mihindukulasooriya, and S. Auer, Ontolearner: A modular python library for ontology learning with large language models, 2026. arXiv: 2607.01977 [cs.AI]. [Online]. Available: https://arxiv.org/abs/2607.01977.

[5] H. B. Giglou, S. Auer, and J. D’Souza, When does bigger help? a controlled study of llm scale for ontology learning, 2026. arXiv: 2608.31118 [cs.AI]. [Online]. Available: https://arxiv.org/abs/2608.31118.

[6] Qwen Team, Qwen3.6-35B-A3B: Agentic coding power, now open to all, Apr. 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.6-35b-a3b.

[7] Z. Nussbaum, J. X. Morris, B. Duderstadt, and A. Mulyar, Nomic embed: Training a reproducible long context text embedder, 2024. arXiv: 2402.01613 [cs.CL].

[8] S. Bannour Souihi and S. Kara, “LIST-CortAIx at LLMs4OL 2026 Task Flagship: Stage-wise Design and Evaluation of a Modular LLM Pipeline for Ontology Construction,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[9] Qwen Team, Qwen3.5: Towards native multimodal agents, Feb. 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.5.

[10] T. B. Brown, B. Mann, N. Ryder, et al., Language models are few-shot learners, 2020. arXiv: 2005.14165 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2005.14165.

[11] P. Lewis, E. Perez, A. Piktus, et al., Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. arXiv: 2005.11401 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2005.11401.

[12] N. Reimers and I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, 2019. arXiv: 1908.10084 [cs.CL]. [Online]. Available: https://arxiv.org/abs/1908.10084.

[13] A. Yang, A. Li, B. Yang, et al., Qwen3 technical report, 2025. arXiv: 2505.09388 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2505.09388.

[14] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, Qlora: Efficient finetuning of quantized llms, 2023. arXiv: 2305.14314 [cs.LG]. [Online]. Available: https://arxiv.org/abs/2305.14314.

[15] X. Zhao and J. Xu, “LaboroAI at LLMs4OL 2026 Tasks Flagship and Reuse: Less is More for LLM-based Ontology Learning,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[16] S. Robertson and H. Zaragoza, “The probabilistic relevance framework: Bm25 and beyond,” Foundations and Trends® in Information Retrieval, vol. 4, pp. 1–174, Sep. 2009. DOI: 10.1561/1500000019.

[17] OpenAI, GPT-5.4, Large language model, accessed June 2026. [Online]. Available: https://openai.com/.

[18] OpenAI, GPT-5.4-mini, Large language model, accessed June 2026. [Online]. Available: https://openai.com/.

[19] Google, Gemini 3.5 Flash, Large language model, accessed June 2026, 2026. [Online]. Available: https://cloud.google.com/vertex-ai/generative-ai/docs/models.

[20] J. Jin, “XuanJi at LLMs4OL 2026 Task Flagship: A Partial Graph-Oriented System for Taxonomy and Typing,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[21] R. Gray, P. Cooper, E. Nayak, et al., “ELLMO at LLMs4OL 2026 Tasks Flagship, Reuse, and Taxonomy: Ontology Learning Across Contexts,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[22] L. McInnes, J. Healy, and S. Astels, “Hdbscan: Hierarchical density based clustering,” The Journal of Open Source Software, vol. 2, p. 205, Mar. 2017. DOI: 10.21105/joss.00205.

[23] Google, Gemma 4 Model Overview, Accessed: September 2026, 2026. [Online]. Available: https://ai.google.dev/gemma/docs/core.

[24] B. Warner, A. Chaffin, B. Clavié, et al., Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. arXiv: 2412.13663 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2412.13663.

[25] Mixedbread, mxbai-embed-large-v1, https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1, Hugging Face model repository, accessed July 13, 2026, 2024.

[26] P. Lisena, J. Plu, O. Moreno Escobar, E. Trouillez, and R. Troncy, “LettRAGraph at LLMs4OL 2026 Task Flagship: Prompt-Guided Ontology Extraction with Ensembling,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[27] N. Mihindukulasooriya, S. Tiwari, C. F. Enguix, and K. Lata, Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text, 2023. arXiv: 2308.02357 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2308.02357.

[28] Gemma Team, Gemma 4 Technical Report, 2026. arXiv: 2607.02770 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2607.02770.

[29] Qwen Team, Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All, Apr. 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.6-35b-a3b.

[30] S. Mishra, D. R. Meena, M. Ojha, K. P. Singh, and K. Singh, “pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[31] Qwen, : A. Yang, et al., Qwen2.5 technical report, 2025. arXiv: 2412.15115 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2412.15115.

[32] U. Bhattacharya, M. de Boer, and S. Sosnovsky, “HybridAI-Squared at LLMs4OL 2026 Task Reuse: Divide and Conquer Does Not Help LLMs Perform Ontology Learning,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[33] L. Liu, Y. Luo, J. Qi, C. Zheng, C. Zhong, and Y. Zhou, “Genuineknowledge-KEOL at LLMs4OL 2026 Tasks Flagship, Reuse, and Taxonomy: Hybrid Training-Free Pipelines for Ontology and Taxonomy Discovery,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[34] Y. Zhang, M. Li, D. Long, et al., Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. arXiv: 2506.05176 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2506.05176.

[35] Anthropic, Claude Opus 4.6, Model documentation, accessed July 16, 2026, 2026. [Online]. Available: https://docs.anthropic.com/en/docs/about-claude/models/overview.

[36] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch, Improving factuality and reasoning in language models through multiagent debate, 2023. arXiv: 2305.14325 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2305.14325.

[37] M. Canal-Esteve, Y. Gutiérrez, J. I. Abreu, and F. Llopis, “SEMA at LLMs4OL 2026 Task C: Continual Pretraining on Serialized Ontologies for Cross-Domain Taxonomy Discovery,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[38] S. Gururangan, A. Marasović, S. Swayamdipta, et al., “Don’t stop pretraining: Adapt language models to domains and tasks,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds., Online: Association for Computational Linguistics, Jul. 2020, pp. 8342–8360. DOI: 10.18653/v1/2020.acl-main.740. [Online]. Available: https://aclanthology.org/2020.acl-main.740/.

[39] Meta, Llama 3.2–3B, Model card, 2024. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.2-3B.

[40] G. Zhao, Y. Tang, and Z. Li, “BigDipperAI at LLMs4OL 2026 Task Flagship: Structured Generation of Primitive Ontology Triples with Schema-Aligned Auxiliary Supervision,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[41] C. Deng, Y. Fan, C. Zeng, and C. Zhang, “VUL337-OL at LLMs4OL 2026 Tasks Flagship and Reuse: Task-Native Constraints for Ontology Construction and Extension,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th ISWC, 2026.

[42] DeepSeek-AI, DeepSeek-V4 Preview Release, https://api-docs.deepseek.com/news/news260424/, Accessed July 23, 2026, 2026.

[43] XiaomiMiMo, MiMo-V2.5-Pro, https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro, Model repository, accessed July 29, 2026, 2026.

[44] E. Ku, “Volt-IQ at LLMs4OL 2026: Constrained Graph Synthesis and Domain-Calibrated Ontology Learning Systems for the Flagship, Reuse, and Taxonomy Tasks,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at the 25th International Semantic Web Conference, 2026.

[45] B. Smith et al., “The OBO Foundry: Coordinated Evolution of Ontologies to Support Biomedical Data Integration,” Nature Biotechnology, vol. 25, no. 11, pp. 1251–1255, 2007. DOI: 10.1038/nbt1346.

[46] L. Cooper, A. Meier, M.-A. Laporte, et al., “The planteome database: An integrated resource for reference ontologies, plant genomics and phenomics,” Nucleic Acids Research, vol. 46, no. D1, pp. D1168–D1180, 2018. DOI: 10.1093/nar/gkx1152.

[47] D. M. Dooley et al., “FoodOn: A Harmonized Food Ontology to Increase Global Food Traceability, Quality Control and Data Integration,” npj Science of Food, vol. 2, no. 1, p. 23, 2018. DOI: 10.1038/s41538-018-0032-6.

[48] M. Vizcaı́no, J. L. Chand, and A. Symonds, “semantic-swingers at LLMs4OL 2026 Tasks Flagship, Reuse, and Taxonomy: Retrieval, Fine-Tuning, and the Seen–Unseen Gap in LLM Ontology Learning,” in LLMs4OL 2026: The 3rd Large Language Models for Ontology Learning Challenge at ISWC 2026, 2026.

[49] E. J. Hu, Y. Shen, P. Wallis, et al., Lora: Low-rank adaptation of large language models, 2021. arXiv: 2106.09685 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2106.09685.

[50] OpenAI, Introducing GPT-4.1 in the API, Apr. 2025. [Online]. Available: https://openai.com/index/gpt-4-1/.

[51] X. Wang, J. Wei, D. Schuurmans, et al., Self-consistency improves chain of thought reasoning in language models, 2023. arXiv: 2203.11171 [cs.CL]. [Online]. Available: https://arxiv.org/abs/2203.11171.

Downloads

Published

2026-11-02

How to Cite

Babaei Giglou, H., Auer, S., & D’Souza, J. (2026). LLMs4OL 2026 Overview: The 3rd Large Language Models for Ontology Learning Challenge. Open Conference Proceedings, 11. https://doi.org/10.52825/ocp.v11i.3728

Funding data