Gradient-Based Adversarial Factual Consistency Evaluation for Abstractive Summarization

Abstract

Neural abstractive summarization systems have gained significant progress in recent years. However, excessive abstractiveness inevitably leads to factual errors, which poses a challenge to the robustness of the systems. In this paper, we proposed an efficient weak-supervised adversarial data augmentation approach to form the factual consistency dataset. Based on the artificial dataset, we train an evaluation model that can not only make accurate and robust factual consistency discrimination but is also capable of making interpretable factual errors tracing by backpropagated gradient distribution on token embeddings. Experiments and analysis conduct on public annotated summarization and factual consistency datasets demonstrate our approach effective and reasonable.