Knowledge-enriched Text Generation Survey, Tutorial and Reading

This repository contains a list of tutorials, papers, codes, datasets, leaderboards on the topic of Knowledge-enhanced text generation. If you found any error, please don't hesitate to open an issue or pull request.

-- We will continue to add and update related papers and codes on this page (Update on Feb 5th, 2022).

-- indicates available code and indicates high citation in recent years.

Survey paper

A Survey of Knowledge-enhanced Text Generation. Wenhao Yu (ND), Chenguang Zhu (Microsoft), Zaitang Li (CUHK), Zhiting Hu (UCSD), Qingyun Wang (UIUC), Heng Ji (UIUC), Meng Jiang (ND). ACM CSUR, 2022.

To the best of our knowledge, our survey is the first work that presents a comprehensive reviewof knowledge-enhanced text generation. It aims to provide NLG researchers a synthesis and pointer to related researches. Our survey also includes a detailed discussion about how NLG can benefit from recent progress in deep learning and artificial intelligence, including technologies such as graph neural network, reinforcement learning, neural topic modeling and so on.

If you find this repository helpful for your work, please kindly cite the following paper.

@article{yu2022survey,
  title={A Survey of Knowledge-Enhanced Text Generation},
  author={Yu, Wenhao and Zhu, Chenguang and Li, Zaitang and Hu, Zhiting and Wang, Qingyun and Ji, Heng and Jiang, Meng},
  journal={ACM Computing Survey (CSUR)},
  year={2022}
}

Tutorials [video]

We have given a tutorial on the topic of Knowledge-enriched Text Generation in EMNLP 2021. It was held from Nov. 7th to Nov. 11th, 2021, in Dominican Republic. Check out the materials [here].

We will also present a tutorial on a broader topic of Knowledge-augmented Methods for NLP in ACL 2022. It will be held from May. 22nd to Nov. 27th, 2022, in Ireland! Check out the materials [here].

Other related surveys

• A Survey of Knowledge Enhanced Pre-trained Models, on arXiv 2021. [pdf]

• A Survey of Natural Language Generation, on arXiv 2021. [pdf]

• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, on arXiv 2021. [pdf]

• Unstructured Text Enhanced Open-Domain Dialogue System: A Survey, in ACM TIST 2021. [pdf]

• A Survey on Retrieval-Augmented Text Generation, on arXiv 2022. [pdf]

Basic NLG papers and codes

(For new learners, some important papers for general NLG/KENLG.)

• [Seq2Seq] Sequence to Sequence Learning with Neural Networks, in NeurIPS 2014. [pdf]

• [SeqAttn] Neural Machine Translation by Jointly Learning to Align and Translate, in ICLR 2015. [pdf]

• [CopyNet] Incorporating Copying Mechanism in Sequence-to-Sequence Learning, in ACL 2016. [pdf]

• [PointerNet] Get To The Point: Summarization with Pointer-Generator Networks, in ACL 2017. [pdf]

• [Transformer] Attention Is All You Need, in NeurIPS 2017. [pdf]

Pretrained language generation models

• [GPT-2] Language Models are Unsupervised Multitask Learners, at OpenAI blog 2019. [pdf] [official code (tf)] [huggingface (torch)]

• [UniLM] Unified Language Model Pre-training for Natural Language Understanding and Generation, in NeurIPS 2019. [pdf] [official code (torch)]

• [BART] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, in NAACL 2020. [pdf] [fairseq (torch)] [huggingface (torch)]

• [T5] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, in JMLR 2020. [pdf] [official code (tf)] [huggingface (torch)]

• [PEGASUS] Pre-training with Extracted Gap-sentences for Abstractive Summarizationr, in ICML 2020. [pdf] [official code (tf)] [huggingface (torch)]

Controllable generation leanrng methods

• [Posterior Regularization] Deep Generative Models with Learnable Knowledge Constraints, in NeurIPS 2018. [pdf]

• [Plug and Play] Plug and Play Language Models: A Simple Approach to Controlled Text Generation, in ICLR 2020. [pdf] [code (torch)] unsup_gen_for_cms_reasoning)

• [Backprop-based Decoding] Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning, in EMNLP 2020. [pdf] [code (torch)]

• [Weakly Supervision] Summarizing Text on Any Aspects: A Knowledge-Informed Weakly-Supervised Approach, in EMNLP 2020. [pdf] [code (torch)]

Knowledge base-enhanced text generation

• [Question Answering] [Generating Natural Answers by Incorporating Copying and Retrieving Mechanisms in Sequence-to-Sequence Learning] , in ACL 2017. [pdf]

• [Question Answering] Natural answer generation with heterogeneous memory, in NAACL 2018. [pdf]

• [Dialogue System] Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems, in ACL 2019. [pdf] [code (torch)]

• [Dialogue System] Global-to-local Memory Pointer Networks for Task-Oriented Dialogue, in ICLR 2019. [pdf] [code (torch)]

• [Dialogue System] Improving Knowledge-aware Dialogue Generation via Knowledge Base Question Answering, in AAAI 2020. [pdf] [code (torch)]

• [Dialogue System] Learning to Select Knowledge for Response Generation in Dialog Systems, in IJCAI 2019. [pdf]

• [Dialogue System] Diverse and Informative Dialogue Generation with Context-Specific Commonsense Knowledge Awareness, in ACL 2020. [pdf] [code (tf)]

• [Dialogue System] TopicKA: Generating Commonsense Knowledge-Aware Dialogue Responses Towards the Recommended Topic Fact, in IJCAI 2020. [pdf] [code (tf)]

• [Content Manipulation] Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation, in AAAI 2020. [pdf] [code (torch)]

• [Content Manipulation] Fact-based Text Editing, in ACL 2020. [pdf] [code]

• [Summarization] Exploring Human-Like Reading Strategy for Abstractive Text Summarization], in AAAI 2019. [pdf]

• [Table-to-text] Describing a Knowledge Base, in INLG 2018. [pdf] [code (torch)]

Knowledge graph-enhanced text generation

• [Dialogue System] Commonsense Knowledge Aware Conversation Generation with Graph Attention, in IJCAI 2018. [pdf] [code1 (tf)] [code2 (torch)]

• [Dialogue System] Knowledge Aware Conversation Generation with Explainable Reasoning over Augmented Graphs, in EMNLP 2019. [pdf]

• [Dialogue System] DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs, in EMNLP 2019. [pdf] [code (tf)]

• [Dialogue System] Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs, in ACL 2020. [pdf] [code (torch)]

• [Dialogue System] GraphDialog: Integrating Graph Knowledge into End-to-End Task-Oriented Dialogue Systems, in EMNLP 2020. [pdf] [code (tf)]

• [Dialogue System] CARE: Commonsense-Aware Emotional Response Generation with Latent Concepts, in AAAI 2021. [pdf]

• [Dialogue System] Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation, in AAAI 2021. [pdf] [code (torch)]

• [Question Answering] Commonsense for Generative Multi-Hop Question Answering Tasks, in EMNLP 2018. [pdf] [code (tf)]

• [Scientific Writing] Text Generation from Knowledge Graphs with Graph Transformers, in NAACL 2018. [pdf] [code (torch)]

• [Scientific Writing] PaperRobot: Incremental Draft Generation of Scientific Ideas, in ACL 2019. [pdf] [code (torch)]

• [Story Generation] Story Ending Generation with Incremental Encoding and Commonsense Knowledge, in AAAI 2019. [pdf] [code (tf)]

• [Story Generation] Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph, in EMNLP 2020. [pdf] [code (torch)]

• [Story Generation] MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models, in EMNLP 2020. [pdf]

• [Story Generation] KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning, in AAAI 2021. [pdf] [code (torch)]

• [Machine Translation] Knowledge Graphs Enhanced Neural Machine Translation, in IJCAI 2020. [pdf]

• [Summarization] Incorporating Commonsense Knowledge into Abstractive Dialogue Summarization via Heterogeneous Graph Networks, in CCL 2021. [pdf]

• [Entity Description] ENT-DESC: Entity Description Generation by Exploring Knowledge Graph, in EMNLP 2020. [pdf] [code]

• [Eassy Generation] A Sentiment-Controllable Topic-to-Essay Generator with Topic Knowledge Graph, in EMNLP findings 2020. [pdf] [data]

Open knowledge graph-enhanced text generation
(Knowledge graph constructed by OpenIE)

• [Question Answering] Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs, in EMNLP 2019. [pdf]

• [Summarization] Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward, in ACL 2020. [pdf] [code (torch)]

• [Summarization] Boosting Factual Correctness of Abstractive Summarization with Knowledge Graph, in NAACL 2021. [pdf]

• [Summarization] Heterogeneous Graph Neural Networks for Extractive Document Summarization, in ACL 2020. [pdf] [code (torch)]

Grounded text-enhanced text generation

• [Dialogue System] A Knowledge-Grounded Neural Conversation Model, in AAAI 2018. [pdf] [data]

• [Dialogue System] Wizard of Wikipedia: Knowledge-Powered Conversational agents, in ICLR 2019. [pdf] [code (torch)]

• [Dialogue System] Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue, in ICLR 2020. [pdf] [code (tf)]

• [Dialogue System] DeepCopy: Grounded Response Generation with Hierarchical Pointer Networks, in SIGDIAL 2019. [pdf]

• [Dialogue System] Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading, in ACL 2019. [pdf] [code (torch)]

• [Dialogue System] RefNet: A Reference-aware Network for Background Based Conversation, in AAAI 2020. [pdf] [code (tf)]

• [Dialogue System] Thinking Globally, Acting Locally: Distantly Supervised Global-to-Local Knowledge Selection for Background Based Conversation, in AAAI 2020. [pdf] [code (torch)]

• [Dialogue System] Knowledge-Grounded Dialogue Generation with Pre-trained Language Models, in EMNLP 2020. [pdf] [code (torch)]

• [Dialogue System] Bridging the Gap between Prior and Posterior Knowledge Selection for Knowledge-Grounded Dialogue Generation, in EMNLP 2020. [pdf] [code]

• [Dialogue System] Difference-aware Knowledge Selection for Knowledge-grounded Conversation Generation, in EMNLP findings 2020. [pdf] [code (torch)]

• [Question Answering] Generating Well-Formed Answers by Machine Reading with Stochastic Selector Networks, in AAAI 2020. [pdf]

• [Summarization] Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization, in ACL 2018. [pdf] [data]

• [Summarization] BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization, in ACL 2019. [pdf] [code (torch)]

• [Commonsense Generation] Retrieval Enhanced Model for Commonsense Generation, in ACL Findings 2021. [pdf]

• [Commonsense Generation] KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning, in EMNLP Findings 2021. [pdf]

• [Paraphrase] Paraphrase Generation by Learning How to Edit from Samples, in ACL 2020. [pdf]

Controllable text generation with rules

• [Story Generation] Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation, in NeruIPS 2021. [pdf]

• [Story Generation] COINS: Dynamically Generating COntextualized Inference Rules for Narrative Story Completion, in ACL 2021. [pdf] [code (torch)]

Topic-enhanced text generation

• [Dialogue System] Topic Aware Neural Response Generation, in AAAI 2017. [pdf]

• [Dialogue System] A Neural TopicalExpansion Framework for Unstructured Persona-oriented Dialogue Generation, in ECAI 2020. [pdf] [code (tf)]

• [Dialogue System] Context-Controlled Topic-Aware Neural Response Generation for Open-Domain Dialog Systems, in Information Processing an Management 2021. [pdf]

• [Summarization] Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization, in EMNLP 2018. [pdf] [code (torch)]

• [Summarization] Topic-Guided Variational Autoencoders for Text Generation, in NAACL 2019. [pdf]

• [Summarization] Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism, in AAAI 2020. [pdf]

• [Summarization] A Topic Augmented Text Generation Model: Joint Learning of Semantics and Structural Features, in COLING 2020. [pdf]

• [Summarization] [Friendly Topic Assistant for Transformer Based Abstractive Summarization], in EMNLP 2020. [pdf] [code]

• [Machine Translation] Topic-Informed Neural Machine Translation, in COLING 2016. [pdf]

• [Machine Translation] Translating with Bilingual Topic Knowledge for Neural Machine Translation, in AAAI 2019. [pdf]

• [Topic Transfer] A Topic Augmented Text Generation Model: Joint Learning of Semantics and Structural Features, in EMNLP 2019. [pdf]

Keyword-enhanced text generation

• [Dialogue System] Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation , in COLING 2016. [pdf] [code (tf)]

• [Dialogue System] Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory, in AAAI 2018. [pdf] [code (tf)]

• [Dialogue System] Generating Responses with a Specific Emotion in Dialog, in ACL 2019. [pdf]

• [Summarization] Guiding Generation for Abstractive Text Summarization based on Key Information Guide Network, in NAACL 2018. [pdf]

• [Summarization] Inferring Search Queries from Web Documents via a Graph-Augmented Sequence to Attention Network, in WWW 2019. [pdf]

• [Summarization] Coherent Comment Generation for Chinese Articles with a Graph-to-Sequence Model, in ACL 2019. [pdf] [code (torch)]

• [Summarization] Keywords-Guided Abstractive Sentence Summarization, in AAAI 2020. [pdf]

Knowledge-enhanced pretraining

• [KG + LM] A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation, in TACL 2020. [pdf] [code (tf)]

• [Commonsense + LM] Abductive Commonsense Reasoning, in ICLR 2020. [pdf] [code (torch)]

• [Table + LM] TAPAS: Weakly Supervised Table Parsing via Pre-training, in ACL 2020. [pdf] [official code (tf)] [video]

• [KG + LM] JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs, in ACL 2021. [pdf]

• [KG + LM] JAKET: Joint Pre-training of Knowledge Graph and Language Understanding, in AAAI 2022. [pdf]

• [KG + Data-to-text pretraining] KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation, in EMNLP 2020. [pdf] [code (torch)]

Acknowledgement

This page is contributed by Wenhao Yu([email protected]) and Qingyun Wang([email protected]).