An Intelligent Chatbot Utilizing BERT Model and Knowledge Graph

SoYeop Yoo, OkRan Jeong


As artificial intelligence is actively studied, it is being applied to various fields such as image, video and natural language processing. The natural language processing, in particular, is being studied to enable computers to understand the languages spoken and spoken by people and is considered one of the most important areas in artificial intelligence technology. In natural language processing, it is a complex, but important to make computers learn to understand a person’s common sense and generate results based on the person’s common sense. Knowledge graphs, which are linked using the relationship of words, have the advantage of being able to learn common sense easily from computers. However, the existing knowledge graphs are organized only by focusing on specific languages and fields and have limitations that cannot respond to neologisms. In this paper, we propose an intelligent chatbotsystem that collects and analyzed data in real time to build an automatically scalable knowledge graph and utilizes it as the base data. In particular, the fine-tuned BERT-based for relation extraction is to be applied to auto-growing graph to improve performance. And, we have developed a chatbot that can learn human common sense using auto-growing knowledge graph, it verifies the availability and performance of the knowledge graph.

Full Text:



Alsubaiee, S., Altowim, Y., Altwaijry, H., Behm, A., Borkar, V., Bu, Y., and Gabrielova, E., “AsterixDB: A scalable, open source BDMS,” Proceedings of the VLDB Endowment, Vol.7, No.14, pp. 1905-1916, 2014.

Athreya, R. G., Ngonga Ngomo, A. C., and Usbeck, R., “Enhancing Community Interactions with Data-Driven Chatbots-The DBpedia Chatbot,” In Companion of the The Web Conference 2018, pp. 143-146, 2018.

Devlin, J., Chang, M. W., Lee, K., and Toutanova, K., “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.

Fellbaum, C., “WordNet: An Electronic Lexical Database,” Cambridge, MA: MIT Press, 1998.

Hyun, Y. J. and Kim, N. G., “Text Mining-based Fake News Detection Using News And Social Media Data,” The Journal of Society for e-Business Studies, Vol.23, No.4, pp. 19-39, 2018.

Ji, G., He, S., Xu, L., Liu, K., and Zhao, J., “Knowledge graph embedding via dynamic mapping matrix,” In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Vol.1, pp. 687-696, 2015.

Lee, D. H. and Kim, K. H., “Web Site Keyword Selection Method by Considering Semantic Similarity Based on Word2Vec,” The Journal of Society for e-Business Studies, Vol.23, No.2, pp. 83-96, 2018.

Lin, Y., Liu, Z., Sun, M., Liu, Y., and Zhu, X., “Learning entity and relation embeddings for knowledge graph completion,” In Twenty-ninth AAAI conference on artificial intelligence, 2015.

Mahdisoltani, F., Biega, J., Suchanek, F. M., “YAGO3: A Knowledge Base from Multilingual Wikipedias,” Conference on Innovative Data Systems Research (CIDR), 2015.

Paulheim, H., “Knowledge graph refinement: A survey of approaches and evaluation method,” Semantic web, Vol.8, No.3, pp. 489-508, 2017.

Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L., “Deep contextualized word representations,” arXiv preprint arXiv:1802.05365, 2018.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I., “Language models are unsupervised multitask learners,” OpenAI Blog, Vol.1, No.8, 2019.

Speer, R., Chin, J., and Havasi, C., “Conceptnet 5.5: An open multilingual graph of general knowledge,” In Thirty-First AAAI Conference on Artificial Intelligence, Feb. 2017.

Tarau, P. and Figa, E., “Knowledge-based conversational agents and virtual storytelling,” In Proceedings of the 2004 ACM symposium on Applied computing, pp. 39-44, 2004.

Wu, W., Li, H., Wang, H., and Zhu, K. Q., “Probase: A probabilistic taxonomy for text understanding,” In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp. 481-492, 2012.

Yang, Y. J., Lee, B. H., Kim, J. S., and Lee, K. Y., “Development of An Automatic Classification System for Game Reviews Based on Word Embedding,” The Journal of Society for e-Business Studies, Vol.24, No.2, pp. 1-14, 2019.

Yoo, S., Song, J., and Jeong, O., “Social media contents based sentiment analysis and prediction system,” Expert Systems with Applications, Vol.105, pp. 102-111, 2018.

Zhang, Y., Peng, Q., and Christopher D. M., “Graph Convolution over Pruned Dependency Trees Improves Relation Extraction,” In Proceeding of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2205-2215, 2018.

Zhang, Y., Zhong, V., Chen, D., Angeli, G., and Manning, C. D., “Position-aware attention and supervised data improve slot filling,” In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 35-45, 2017.


  • There are currently no refbacks.