FIGURE 1 - available via license: Creative Commons Attribution 4.0 International
Content may be subject to copyright.
Source publication
The knowledge graph completion (KGC) task aims to predict missing links in knowledge graphs. Recently, several KGC models based on translational distance or semantic matching methods have been proposed and have achieved meaningful results. However, existing models have a significant shortcoming–they cannot train entity embedding when an entity does...
Contexts in source publication
Context 1
... y ∈ R K denotes the output probability vector for a prediction for which entity the [MASK] token should be predicted among overall entities and W e ∈ R K×H denotes a learnable parameter matrix. To train the model, we use the cross-entropy loss function: Figure 1 shows the overall process of the baseline method for the case of the head-batch (Figure 1(a)) and tail-batch (Figure 1(b)). ...
Context 2
... y ∈ R K denotes the output probability vector for a prediction for which entity the [MASK] token should be predicted among overall entities and W e ∈ R K×H denotes a learnable parameter matrix. To train the model, we use the cross-entropy loss function: Figure 1 shows the overall process of the baseline method for the case of the head-batch (Figure 1(a)) and tail-batch (Figure 1(b)). ...
Context 3
... y ∈ R K denotes the output probability vector for a prediction for which entity the [MASK] token should be predicted among overall entities and W e ∈ R K×H denotes a learnable parameter matrix. To train the model, we use the cross-entropy loss function: Figure 1 shows the overall process of the baseline method for the case of the head-batch (Figure 1(a)) and tail-batch (Figure 1(b)). ...
Similar publications
In this work, we study the representation space of contextualized embeddings and gain insight into the hidden topology of large language models. We show there exists a network of latent states that summarize linguistic properties of contextualized representations. Instead of seeking alignments to existing well-defined annotations, we infer this lat...
Good data augmentation is one of the key factors that lead to the empirical success of self-supervised representation learning such as contrastive learning and masked language modeling, yet theoretical understanding of its role in learning good representations remains limited. Recent work has built the connection between self-supervised learning an...
In recent years, pre-trained language models (e.g., BERT and GPT) have shown the superior capability of textual representation learning, benefiting from their large architectures and massive training corpora. The industry has also quickly embraced language models to develop various downstream NLP applications. For example, Google has already used B...
Word embeddings are so significant today that it is common to see their application in multiple natural language tasks. Indeed, word embeddings as the first layer of a deep learning model are widely adopted and they can also be found in multiple natural language tasks such as classification of texts and named entity recognition. The focus of this p...
Semantic 3D scene understanding is a problem of critical importance in robotics. While significant advances have been made in simultaneous localization and mapping algorithms, robots are still far from having the common sense knowledge about household objects and their locations of an average human. We introduce a novel method for leveraging common...
Citations
... It learns the structural information of triplets while retaining some of the knowledge from BERT to learn better knowledge graph embeddings. Inspired by masked language modeling (MLM), MEM-KGC [20] is proposed by BONGGEUN CHOI et al. This method first masks the tail entity and treats the head entity and relation as the context for the tail entity. ...
Knowledge graph completion (KGC) utilizes known knowledge graph triples to infer and predict missing knowledge, making it one of the research hotspots in the field of knowledge graphs. There are still limitations in generating high-quality entity embeddings and fully understanding the contextual information of entities and relationships. To overcome these challenges, this paper introduces a novel pre-trained language model-based method for knowledge graph completion that significantly enhances the quality of entity embeddings by integrating entity categorical information with textual descriptions. Additionally, this method employs an innovative multi-layer residual attention network in combination with PLMs, deepening the understanding of the joint contextual information of entities and relationships. Experimental results on the FB15k-237 and WN18RR datasets demonstrate that our proposed model significantly outperforms existing baseline models in link prediction tasks.
... As shown in Figure 1, textual entity descriptions encompass a wealth of information. The descriptionbased KGC methods (Choi et al., 2021;Yao et al., 2019;Wang et al., 2022b) proposed fine-tuning the pre-trained language models (PLMs) to represent the entities and relations based on textual con- Figure 2: (a) The description-based methods use PLM to convert head (X h ) and relation (X r ) to text embeddings. (b) The structure-based methods represent the head index (I h ) and the relation index (I r ) as embeddings to learn the structural information. ...
Knowledge Graph Completion (KGC) aims to conduct reasoning on the facts within knowledge graphs and automatically infer missing links. Existing methods can mainly be categorized into structure-based or description-based. On the one hand, structure-based methods effectively represent relational facts in knowledge graphs using entity embeddings. However, they struggle with semantically rich real-world entities due to limited structural information and fail to generalize to unseen entities. On the other hand, description-based methods leverage pre-trained language models (PLMs) to understand textual information. They exhibit strong robustness towards unseen entities. However, they have difficulty with larger negative sampling and often lag behind structure-based methods. To address these issues, in this paper, we propose Momentum Contrast for knowledge graph completion with Structure-Augmented pre-trained language models (MoCoSA), which allows the PLM to perceive the structural information by the adaptable structure encoder. To improve learning efficiency, we proposed momentum hard negative and intra-relation negative sampling. Experimental results demonstrate that our approach achieves state-of-the-art performance in terms of mean reciprocal rank (MRR), with improvements of 2.5% on WN18RR and 21% on OpenBG500.
... MTL-KGC [51] encoders the text sequences to predict the possibility of the tuples. MEMKGC [52] predicts the masked entities of the triple. StAR [53] utilizes Siamese textual encoders to separately encode the entities. ...
The advent of large language models marks a revolutionary breakthrough in artificial intelligence. With the unprecedented scale of training and model parameters, the capability of large language models has been dramatically improved, leading to human-like performances in understanding, language synthesizing, and common-sense reasoning, etc. Such a major leap-forward in general AI capacity will change the pattern of how personalization is conducted. For one thing, it will reform the way of interaction between humans and personalization systems. Instead of being a passive medium of information filtering, large language models present the foundation for active user engagement. On top of such a new foundation, user requests can be proactively explored, and user's required information can be delivered in a natural and explainable way. For another thing, it will also considerably expand the scope of personalization, making it grow from the sole function of collecting personalized information to the compound function of providing personalized services. By leveraging large language models as general-purpose interface, the personalization systems may compile user requests into plans, calls the functions of external tools to execute the plans, and integrate the tools' outputs to complete the end-to-end personalization tasks. Today, large language models are still being developed, whereas the application in personalization is largely unexplored. Therefore, we consider it to be the right time to review the challenges in personalization and the opportunities to address them with LLMs. In particular, we dedicate this perspective paper to the discussion of the following aspects: the development and challenges for the existing personalization system, the newly emerged capabilities of large language models, and the potential ways of making use of large language models for personalization.
... Instead of encoding the full text of a triple, many works introduce the concept of Masked Language Model (MLM) to encode KG text ( Fig. 18(b)). MEM-KGC [145] uses Masked Entity Model (MEM) classification mechanism to predict the masked entities of the triple. The input text is in the form of Similar to Eq. 4, it tries to maximize the probability that the masked entity is the correct entity t. ...
Large language models (LLMs), such as ChatGPT and GPT4, are making new waves in the field of natural language processing and artificial intelligence, due to their emergent ability and generalizability. However, LLMs are black-box models, which often fall short of capturing and accessing factual knowledge. In contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are structured knowledge models that explicitly store rich factual knowledge. KGs can enhance LLMs by providing external knowledge for inference and interpretability. Meanwhile, KGs are difficult to construct and evolving by nature, which challenges the existing methods in KGs to generate new facts and represent unseen knowledge. Therefore, it is complementary to unify LLMs and KGs together and simultaneously leverage their advantages. In this article, we present a forward-looking roadmap for the unification of LLMs and KGs. Our roadmap consists of three general frameworks, namely, 1) KG-enhanced LLMs, which incorporate KGs during the pre-training and inference phases of LLMs, or for the purpose of enhancing understanding of the knowledge learned by LLMs; 2) LLM-augmented KGs, that leverage LLMs for different KG tasks such as embedding, completion, construction, graph-to-text generation, and question answering; and 3) Synergized LLMs + KGs, in which LLMs and KGs play equal roles and work in a mutually beneficial way to enhance both LLMs and KGs for bidirectional reasoning driven by both data and knowledge. We review and summarize existing efforts within these three frameworks in our roadmap and pinpoint their future research directions.
... The triple-based baselines include RESCAL (Nickel et al., 2011), TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), Com-plEx (Trouillon et al., 2016), RotatE (Sun et al., 2019), TuckER (Balažević et al., 2019), HAKE (Zhang et al., 2020a), CompGCN (Vashishth et al., 2019), and HittER (Chen et al., 2021). The textbased baselines 1 include Pretrain-KGE (Zhang et al., 2020b), KG-BERT (Yao et al., 2019), StAR (Wang et al., 2021a), and MEM-KGC (Choi et al., 2021). The LLM-based baselines are based on ChatGPT. ...
... The objective of KG completion is to predict the missing entities or relations given any two elements in a (head entity, relation, tail entity) triple. Among various models for KG completion, KG embedding, or learning a distributed KG representation, has demonstrated great power over the past years [4]- [21]. With the help of these models, the entities and relations of KG are embedded in a continuous embedding space. ...
Extra information, such as hierarchical entity types, entity descriptions or some text corpus are recently used to enhance Knowledge Graph Completion (KGC). A typical task in this setting is building entities’ description information into some embedding models. Existing approaches under this task usually use simple embedding models, which have difficulty in handling the complex structures of the knowledge graphs. These models are also limited in the way where description representation is combined with structure representation, which requires an impractical large set of weight parameters increasing in proportion to the number of entities in the knowledge graph. This paper aims at developing more effective embedding models that jointly represent the structure information of the knowledge base with the description of entities. We propose more principled approaches named Dimensional Attentive Combination (DAC) for the composition of structure representation and description representation with fixed-size parameters independent of entity amount, and the composition builds upon more powerful knowledge graph embedding models. The proposed model significantly reduces the weight parameters and can extend to KGs with a large set of entities or involving sparse data. Experimental comparison on link prediction and relation prediction shows that our approaches, even under a simple description-encoding model, improve upon the baselines by a significant margin.
The advent of large language models marks a revolutionary breakthrough in artificial intelligence. With the unprecedented scale of training and model parameters, the capability of large language models has been dramatically improved, leading to human-like performances in understanding, language synthesizing, common-sense reasoning, etc. Such a major leap forward in general AI capacity will fundamentally change the pattern of how personalization is conducted. For one thing, it will reform the way of interaction between humans and personalization systems. Instead of being a passive medium of information filtering, like conventional recommender systems and search engines, large language models present the foundation for active user engagement. On top of such a new foundation, users’ requests can be proactively explored, and users’ required information can be delivered in a natural, interactable, and explainable way. For another thing, it will also considerably expand the scope of personalization, making it grow from the sole function of collecting personalized information to the compound function of providing personalized services. By leveraging large language models as a general-purpose interface, the personalization systems may compile user’s requests into plans, calls the functions of external tools (e.g., search engines, calculators, service APIs, etc.) to execute the plans, and integrate the tools’ outputs to complete the end-to-end personalization tasks. Today, large language models are still being rapidly developed, whereas the application in personalization is largely unexplored. Therefore, we consider it to be right the time to review the challenges in personalization and the opportunities to address them with large language models. In particular, we dedicate this perspective paper to the discussion of the following aspects: the development and challenges for the existing personalization system, the newly emerged capabilities of large language models, and the potential ways of making use of large language models for personalization.
Large language models (LLMs), such as ChatGPT and GPT4, are making new waves in the field of natural language processing and artificial intelligence, due to their emergent ability and generalizability. However, LLMs are black-box models, which often fall short of capturing and accessing factual knowledge. In contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are structured knowledge models that explicitly store rich factual knowledge. KGs can enhance LLMs by providing external knowledge for inference and interpretability. Meanwhile, KGs are difficult to construct and evolve by nature, which challenges the existing methods in KGs to generate new facts and represent unseen knowledge. Therefore, it is complementary to unify LLMs and KGs together and simultaneously leverage their advantages. In this article, we present a forward-looking roadmap for the unification of LLMs and KGs. Our roadmap consists of three general frameworks, namely,
1) KG-enhanced LLMs,
which incorporate KGs during the pre-training and inference phases of LLMs, or for the purpose of enhancing understanding of the knowledge learned by LLMs;
2) LLM-augmented KGs,
that leverage LLMs for different KG tasks such as embedding, completion, construction, graph-to-text generation, and question answering; and
3) Synergized LLMs + KGs
, in which LLMs and KGs play equal roles and work in a mutually beneficial way to enhance both LLMs and KGs for bidirectional reasoning driven by both data and knowledge. We review and summarize existing efforts within these three frameworks in our roadmap and pinpoint their future research directions.
